Verification Is All You Need

The second job nobody warned me about

Back then, my workflow with AI agents looked like this: I’d ask the agent to build a feature, it would finish, and then I’d open the app to find out what didn’t work.

That last part was the problem. Explaining a failure back to the agent was often harder than asking for the feature in the first place. Something felt wrong, but I didn’t always know which detail mattered. So I’d describe what I saw, the agent would change something, and I’d click through the same flow again. And again.

Faster coding had quietly handed me a fairly tedious second job: QA tester for my own AI. The agent’s time was cheap. Mine wasn’t, and the loop was running on mine.

What I actually wanted sounds simple: less work for me, but I can still trust the agent. Most people treat those as a trade-off. Check everything yourself and you can trust it, but you’re exhausted. Let it go and it’s easy, but you’re nervous. I didn’t want to pick one.

The agent wasn’t lying

It’s tempting to say “the AI lies about what it tested.” That’s not what I saw. The agent ran the tests. The build passed. The page loaded. Every claim in its report was true.

The problem was that what it checked wasn’t enough to guarantee the task was finished. There’s a gap between “what I verified” and “this is actually done,” and the agent lived on the wrong side of it. That’s not an honesty problem. It’s a coverage problem.

And it’s sneakier than lying, because nothing in the report is false. You read it, it all checks out, you nod. Then you open the app and the data never shows up in the UI, even though the API “returns 200.”

Looking back, that’s exactly what my manual testing was. I was checking against what I meant. The agent was checking against something narrower. Every time I opened the app, I was closing that gap by hand, slowly, one bug report at a time.

So “make the agent verify its work” isn’t the fix on its own. An agent can verify diligently forever and still miss, if it’s verifying the wrong things. Two pieces were missing: knowing what done actually means, and being able to observe it.

”Done” lives in your head

The person who knows what done looks like is the human. The catch: that knowledge is buried. It’s tacit, made of intuition, old bugs, and things you’d notice in two seconds of clicking around but would never think to write down.

Ask me “what are the acceptance criteria?” and you’ll get a shrug and half a list. That’s not laziness. Humans are bad at recalling what they know. But put something concrete in front of them, and they can immediately say “right” or “wrong” and start going from there.

“Done means you can add an item, refresh, and it’s still there, right?”

“Yes, and it should be sorted by date. And the empty state shouldn’t look broken.”

I’d never have recalled those last two on my own. I recognized them in about a second.

So the agent’s job flips. It shouldn’t wait for a perfect spec, and it shouldn’t ask open-ended questions either. Before writing any code, it should propose a concrete definition of done: scenarios, example inputs and outputs, what the user will see. Then I react. A guess that’s wrong in a specific way is useful, because it gives me something to push against. A few minutes of reacting up front costs far less than an hour of discovering the gaps afterward.

Fun fact: this post was written exactly this way. I knew roughly what I wanted to say, but not how. So I had an AI interview me, propose outlines, and let me say “no, not that” until the real shape came out.

Give the agent eyes

A perfect definition of done is useless if the agent can’t observe it. An agent can only verify what it can see. A test suite is one pair of eyes, and it’s often pointed at the wrong place: the unit tests pass while the screen stays empty.

The tricky part is that every task needs different eyes:

• A web UI needs a browser the agent can drive, screenshots, and console logs.
• A physics game needs state dumps. Did the castle actually fall? Where did the projectile land?
• A data pipeline needs reconciliation. Do the totals balance against the source?
• A CLI tool needs to be run for real, with the output read back.

There’s no universal verification tool. Part of the job is giving the agent the right instrument for this task. Usually that’s something small: a script, a debug endpoint, a headless mode. And it pays off well past the first use, because once it exists, every future task in that area becomes cheaper to trust.

A good rule of thumb: verify the way the user experiences it. End to end, not just the pieces in isolation.

The secret sauce: verification

Put those two together, an agreed definition of done and the instruments to observe it, and something changes. The agent now has its own finish line, and a way to tell whether it crossed it.

Without that, an agent stops at the first plausible point and reports success. Honestly. And you’re back to babysitting.

With it, the agent can run its checks, find the gap, fix it, and check again, looping without needing you. This is the secret sauce that lets an agent cook for hours. You start it, go do something else (or finally sleep), and when you come back, a working product is sitting there waiting for you. No babysitting once it’s started.

My job changed too. I went from manual tester to the person who confirms what done means and makes sure the agent can see it. The effort moved to the front: a few minutes of reacting to proposals, then hands off.

Less work for me, and I can trust the result. Turns out it was never really a trade-off.

Steal this

Here’s a starting prompt to add before any non-trivial task. Tweak it to fit your setup.

Before writing any code:
1. Propose a concrete definition of done: specific scenarios a user
   would go through, with the expected result for each. Make your
   best guess. I'll correct it.
2. For each scenario, say how you will verify it yourself: what you'll
   run and what you'll look at. If you can't observe something, say so
   and propose the smallest tool that would let you.
Wait for my confirmation.

After implementing:
- Run every check. Don't report done until each scenario passes.
- For each scenario, show the evidence (output, screenshot, log),
  not just a claim.
- If a check reveals a gap, fix it and re-run without asking me.

The agent was never lying. It just needed to know where the finish line was, and a way to see it.