Perception is selective. Proxies are stand-ins. The gap between what a system was built to do and what we’ve decided it should do is where things quietly go wrong.
You’ve seen the argument. We only perceive a sliver of the electromagnetic spectrum. The brain filters everything. So reality as we know it is an illusion, and if you could see the real thing you’d probably go mad.
It’s a good line. Travels well. The last step just doesn’t follow from the first two.
Nobody claims our senses are comprehensive. We can’t see infrared. We can’t hear ultrasound. A dog’s nose beats ours without trying. But that’s about range, not accuracy. A bat maps the world with sound and catches moths in the dark doing it. That world is real to the bat. Ours is real to us.
Three questions keep getting tangled together. What exists. What we can detect. What we end up experiencing. Answering one doesn’t answer the others.
On the first two, the science is boringly clear. Perception is selective. The brain does a lot of work building what you experience. Neither of those facts gets you to “therefore reality is fake.” That’s a leap, and it’s usually made by someone who likes how it sounds.
Here’s the thing. Perception evolved to keep us alive. Then we turned around and asked it to reveal the fundamental nature of existence. Those were never the same job. Getting annoyed at a hammer for not being a microscope isn’t the hammer’s fault.
The intent gap is the distance between what a system was built to do and what we’ve decided it should do.
Once you see it, you find it everywhere.
| Proxy | Meant to measure | Now optimised as | What hollows out |
|---|---|---|---|
| Test scores | Learning | The goal itself | Schools teach to the test |
| Engagement metrics | Value | The target | Platforms optimise for outrage because outrage keeps people scrolling |
| Stock price | Company health | Quarterly numbers | Decisions that hollow the company out over five years |
In each case the proxy was reasonable at the start. Then it got optimised past the point of usefulness, and nobody noticed, because the number kept going up.
AI coding assistants make writing code cheap. That’s the pitch and it’s true. What doesn’t follow is that correctness got cheap too. Those are separate problems, and only one of them is being solved.
Watch how it plays out. You ask a model to write a function. It writes one. Looks fine. You ask the same model whether it’s fine. It says yes.
You’ve got a system grading its own homework, and it passes itself every time, because the thing that made it good at writing plausible code is the same thing that makes it bad at spotting the flaws in plausible code.
Plenty of teams run multi-agent setups to fix this. Architect, review, dev, review, final gate. Different models, different temperature settings, different prompts. It helps a bit. Different models make different mistakes, so you catch more of them.
But every stage still agrees on what success looks like. Nobody’s job is to attack the definition of success itself. So the same blind spot survives the whole pipeline and gets waved through at the gate.
The survey numbers keep landing in the same place. Most developers don’t fully trust AI-generated code. A large chunk of them don’t check it properly anyway. A meaningful slice of it ships with security holes. The tooling got faster. The assurance didn’t.
You can’t fix this by adding more of the same. Another reviewer with the same assumptions is another stamp on the same document.
What you want is a specialist with the opposite objective. One model builds. Another gets rewarded for finding reasons the build should be rejected. Not “does this satisfy the requirements” but “what does the requirement assume, and what happens when that assumption is wrong.” Not vague risk lists, which are easy to generate and useless to act on. Specific failure hypotheses you can actually test.
That’s a harder thing to train. Reward a model for finding problems and it will invent problems. Reward it for finding real problems and you need a way to verify that a problem is real, relevant, and new. That verification step is the whole difficulty.
A pipeline of models that all think alike, all checking each other’s work, all agreeing everything’s fine.
The proxies aren’t the problem. We need them. You can’t measure learning or code quality or value directly, so you measure something correlated and hope it tracks. That’s how it’s always worked.
The problem is forgetting. Forgetting that the number was a stand-in. Forgetting that “the model wrote code” isn’t the same as “the code is correct.” Forgetting that “the reviewer approved it” isn’t the same as “it’s safe.”
The gap doesn’t close on its own. It widens quietly, because the numbers keep going up and nobody wants to be the one asking what they’re actually measuring.
The intent gap is not a failure of intelligence. It is a failure of attention. The proxy was always a stand-in. The map was never the territory. The metric was never the goal.
Essay · October 2026