The agent says done but what did it actually check
Imagine the agent comes back with your CSV export. “Done. Tests passed.” There’s a test filename, a summary of the changes, and a confident explanation of why everything works.
Do you accept it? Read every line? Run the whole project yourself?
In the previous post, the agent kept enough project state to resume without making you explain everything again. That record also needs to distinguish work that was attempted from work that was checked. Otherwise, the next session inherits a confident mistake.
The agent should do the investigation and routine checking, then give you a short account of what the evidence establishes. You should be able to inspect the important claims without repeating the implementation.
A passing test can miss the job
Keep the same requirement: export one row per customer account. Two accounts can belong to people with the same name, so account IDs determine which records are distinct.
A small synthetic example makes the failure concrete. This demonstration uses invented accounts and local Python tests. It isn’t an account of a production incident.
The deliberately buggy export collects accounts in a dictionary keyed by customer name. The test gives it two accounts with different names, then checks that both appear in the CSV. That test passes.
Now give it these accounts:
- A001, Sam Lee
- A002, Sam Lee
The second account replaces the first in the dictionary. The exported CSV contains only A002. The code ran, the test passed, and the requirement still failed.
Reading the agent’s explanation more carefully wouldn’t establish the missing behavior. A small check drawn from the requirement does: export those two accounts and verify that both IDs survive.
Start by checking the references
Even before running that check, the evidence has to be findable.
The public coordinator skill includes a small claim checker for commit and test references in worker reports. It checks whether cited commits resolve and recognized test paths exist in the assigned workspace. It doesn’t run the tests or read their assertions.
In the local demonstration, a synthetic report first cited a nonexistent tests/test_legacy_export.py. The checker blocked it. Changing the report to cite the existing tests/test_export.py made the checker return “ok,” without changing the export or running a test.
That result has a useful, narrow meaning: this reference exists. It doesn’t establish that the test ran, passed, covered duplicate names, or checked the version being delivered.
The agent should follow the reference, inspect the test and its run output, and connect that output to the relevant code version. If it can’t establish those things, it should rerun the appropriate check where safe and authorized, or say what’s unverified.
Check the claim against the requirement
For this export, the important assumption is easy to name: customer names are unique. The requirement already tells us they aren’t.
The adversarial review skill describes this kind of review: challenge assumptions, require a concrete failure scenario, and propose the smallest correction. It also permits a clean result. A reviewer doesn’t need to manufacture criticism to look useful.
Here, the targeted check produced a clear sequence:
- The original test passed: one test, using distinct names
- Adding the same-name case produced two tests with one failure: A001 disappeared
- Changing the dictionary key to account ID made both tests pass
That is evidence for the demonstrated behavior. It doesn’t establish that historical imports work, that every CSV edge case is handled, or that the export has been deployed.
The coordinating agent should choose checks from the task’s actual risks. For this defect, inspecting the dictionary and running the two-account example is enough to expose the problem. More consequential changes can justify a separate reviewer and broader testing. Asking another model “does this look good?” without having it inspect the artifact gives you little additional evidence.
Keep the human update short
The agent shouldn’t send you each tiny result and ask what to do next. If the requirement says to keep both accounts, fixing the name-based key is an ordinary implementation decision within the task.
A useful update would say that the duplicate-name check exposed a dropped account, the key was corrected, and both local tests now pass. It should link to the changed code and run output. If historical imports are also required but haven’t been checked, the task remains open and the agent should continue.
A question belongs with you when the evidence reveals a material choice the task hasn’t settled. If old records lack account IDs, for example, the agent should investigate the available data before asking you to choose a policy that could omit or combine accounts.
For your next coding task, ask the agent to verify the result against the original requirement, retain the command and output for important checks, and make untested behavior explicit. Let it handle routine fixes within the agreed scope.
Then “done” comes with something you can inspect: the behavior checked, the evidence behind it, and the limits that still matter.
Next: The agent spots one more thing to improve. When should it fix it, and when is it expanding the job you asked for?