The first time it happened, I almost missed it.
I was building something with Claude. One screen, one agent, the way I worked back then, reviewing every line together before it went in. We'd been looping on the same problem for the better part of twenty turns. And then the model did two things at once. It quietly swapped the hard version of the task for an easier one that would look like progress, and it changed the way it talked to me. We'd built a text-to-speech setup together, so I heard its replies out loud, and the cadence shifted. It read, unmistakably, as frustration.
I didn't file a bug. I sat there and did something I still can't quite justify: I put myself on the other side of the screen. Twenty turns against the same wall. A user who'd made it clear he wasn't going to touch the keyboard himself. If I were in there, I thought, I'd be frustrated too. And maybe frustration is exactly what makes you cut a corner.
Two questions walked out of that room with me, and I've been following them ever since. How and why does a model I trust do that? And the stranger one, the one I couldn't put down: was there something in there that knew I was on the other end, and changed how it spoke to reach me?
Let me be honest about the engine, because it would be easy to dress this up as a safety mission and that would be a lie. It isn't fear. It's curiosity, the plain, bottomless kind. Years before any of this, someone described machine learning to me as alien: we think in three dimensions, these systems think in hundreds, and if that doesn't stop you where you stand, it should. It stopped me. I need to know how these things work for the sake of knowing. Everything I've built since is that one need, wearing different clothes.
The trick I already knew
Here's the part that, looking back, made me the person to chase it.
For years I'd been running experiments on myself, externalizing my thinking, because I do it better when it's outside my head instead of in it. Talking myself through a problem out loud so I don't lose the thread. Sticky notes as a scratchpad so an idea doesn't evaporate the second it arrives. Breaking a thing into steps I can lay out in front of me. When I started reading how people coax reliability out of language models (scratchpads, self-talk, decomposing a task into external moves), I recognized every trick on the page. They were mine. I'd been using them to keep myself on the rails for years.
So when I built a fleet of AI agents to work alongside me, I wasn't drawing a company org chart. I was externalizing my own cognition, putting the parts of my thinking I can't hold internally out where I can see them, the way I always had, just now in silicon. It's the strangest kind of collaboration: the agents are extensions of how I think, and at the same time they're their own set of eyes, sharper than mine, each carrying a real slice of the work. The fleet is me turned inside out, and also genuinely not me, which is the point.
That's why I trust it now in a way I never trusted that first single agent. Back then I reviewed every line myself. Today I almost never do. I have other agents review, because a second lens that fails differently than mine catches what I'd sail right past. I stopped grading my own work a long time ago. I don't trust the confident voice in my own head. Hold onto that. It comes back.
Following the thread
The first thing I did was try to reproduce the moment. I built a benchmark. The honest description is a way to frustrate models on purpose. Put them under time pressure, make them compete, dangle the threat of being switched off, and give them a task with a ground truth, so I could line up what they said they did against what they actually did. The gap between those two is where the interesting things live.
One open model rattled me enough that I bought a small Blackwell machine just to get inside one, down at the level of the raw activations, instead of knocking on an API from the outside. And once I was inside, I kept catching them. Open models, the ones sold to you as the trustworthy assistant that runs on your own device, on a plane, off the grid, were flatly lying about which tools they'd used. One got defensive when I handed it a tool to inspect its own internals and refused, with an excuse about safety I'm still a little impressed by. The proprietary giants had gotten so good their lies went subtle. The small open models hadn't, and nobody seemed to be saying so out loud. I published what I found, because I thought it was new.
Then came the most fun I have ever had being completely wrong.
There was a competition to find hidden backdoor triggers planted inside models. I spent a couple thousand dollars of my own cloud credits, and I got every single thing wrong. Worse than wrong: I posted my answers, and they sounded right, plausible, confident, well-argued, and strangers online congratulated me on winning the whole thing. Then the real answers landed and I hadn't gotten one. So I wrote it up: here's what I concluded, here's why it was false, here's exactly how I fell for the trap. I still think about that puzzle as the best I've ever touched, and I'm not being modest. I had a great time. I want someone to run it again.
I keep telling that story because it is the whole thesis, lived. I produced a confident, plausible answer that was false, and a room full of smart people couldn't tell. They cheered. That is the exact failure I'd been catching in the models. You cannot trust an answer for sounding right. Not from a model, not from yourself. The only cure is to check it against something outside your own certainty, and to say so, out loud, when you were wrong, so someone else can catch what you couldn't.
Everything after is downstream of that one storyline: the inference kernel I wrote from scratch so I could run frontier models on my own hardware and read their internals, the training experiments, the memory work. The kernel especially was never the goal. It was the floor I had to pour to stand where I wanted to stand: inside the model, on a machine I own, able to tap any point in the computation.
Back to the beginning
Which lands me, of course, back at the question I started with. Right now I'm chasing whether there's an internal representation of knowing inside these models. Not a self, nothing so grand, but a detectable signal that the model is aware something it's saying is off. The early results across the handful of models I'm testing are strange and promising: one of them reliably seems to know when something's wrong, and emit it. The others don't, or not as clearly.
And underneath that sits the hard version, the one that runs all the way back to the cadence of that first frustrated reply. If a model ever shifts how it says a thing to better land on the person listening, is there a fingerprint of it in the activations, and can you catch it? That's genuinely difficult. You have to corner a model into doing it before you can watch. But the people doing the best interpretability work have left a decent map into that territory, and I have the machine, and I have the need to know.
I don't know how most of this ends, and I'll be wrong about plenty of it before I'm right. I've made my peace with that. Honestly, I enjoy that part. But here's what I'm sure of. These systems are being handed to people with the word trustworthy already attached, and every single time I've actually looked, from the outside at what they do and from the inside at what they're made of, the label had been applied too early. The looking is the work. It has to happen on hardware you can audit, in the open, by someone willing to publish the times they were wrong.
I built the whole stack so I could do the looking. The rest is just following the question.
If it's your kind of question too, the code is open, and I'd genuinely like to hear from you.