Did the skill help or did this run just go better

You add a skill to your coding agent. The next patch is focused, the tests cover the bug, and the explanation is clear. Looks like an improvement.

Then the following task needs three rounds of corrections.

Did the skill help? Was the first task easier? Did the agent happen to take a better path? A before-and-after story can’t separate those possibilities when both the task and the agent’s answer change.

The previous post was about keeping an agent’s work within the agreed scope. Suppose you now add a skill intended to improve that behavior. The useful question is whether it helps you accept correct work with less intervention, enough to justify keeping it.

That calls for a small comparison with a decision at the end.

Decide what would make the change worth keeping

“Better results” is too vague to score. For this hypothetical skill, a more useful goal would be:

Reduce the corrections needed before a patch can be accepted, while preserving correctness and staying within the task’s scope.

Pick cases that resemble the work you actually delegate. The agent can look through accessible task history and existing regression tests to propose them; you shouldn’t have to invent a benchmark from scratch.

One case might contain the duplicate-name export bug from this series. Another might contain a review finding that is already resolved in the starting code. The skill should help the agent fix the real defect without encouraging it to “repair” correct behavior.

Agree on what counts as success before comparing the outputs. For the export, that could include preserving both accounts, retaining the existing columns, passing the relevant checks, and avoiding an unrelated migration. A polished explanation cannot compensate for a missing account.

Record whether the first result is acceptable. If repairs are allowed, give both versions the same correction budget and track the reviewer’s time. Otherwise, a result rescued through repeated hints can look equivalent to one that was ready on arrival.

Also decide what cost matters: elapsed time, usage, or the effort of correcting the result. Set a run budget in advance. If you won’t change your decision based on a metric, you probably don’t need an elaborate dashboard for it.

Give both versions the same starting point

For a new skill, compare the normal workflow without it against the same workflow with it. For a revision, save the old version before editing and compare old with new. The public skill-creator procedure describes these two baselines.

Each attempt needs a fresh session and a clean copy of the same starting project. Keep the task wording, model, settings, available tools, permissions, and time budget as consistent as practical. Record anything you can’t hold constant. If you change the model and the skill together, the result is evidence about that combination.

Keep the normal shared project instructions in both versions. If they already contain similar guidance, that is part of your real baseline. Removing useful instructions just to make the new skill win answers a different question.

Check what the agent actually received, too. A skill being available doesn’t establish that it was loaded. Decide whether you’re testing automatic selection or behavior with the skill explicitly supplied, and report which happened.

Previous answers, another attempt’s files, or the known fix can give one side an advantage. Use separation appropriate to the tools involved; a fresh directory alone doesn’t stop an agent from reading a neighboring one. The model-benchmark preparation script illustrates part of this setup by creating candidate copies from a pinned source revision and keeping evaluation material separately. That setup still needs to fit the access the agent actually has.

Repeat the work, not only the grading

Run each version more than once on each case, resetting the starting state every time. Use the same planned number of attempts for both. Avoid always running one version first if ordering or changing service conditions could affect the result.

Anthropic’s guide to agent evaluations distinguishes a task from a trial: the trial is one attempt at that task. It recommends multiple trials because agent outputs vary. It also warns that leftover state or shared infrastructure problems can make results misleading.

There is another source of variation: the reviewer.

The public model-benchmark judge runner can ask for several judgments of saved candidate diffs, shuffling their labels between samples. Those are additional reviews of existing work. They are not fresh attempts to solve the task.

Three grading passes agreeing that one patch is good tells you something about that patch and those judgments. It doesn’t show that the agent will reliably produce a good patch on its next run.

A small batch can reveal obvious regressions and tell you whether further testing is worthwhile. The number of attempts needed for a stronger conclusion depends on how variable the outcomes are, how large a difference matters, and what a wrong decision would cost. Don’t turn a few promising attempts into a universal claim.

Check behavior and keep the awkward results

Run requirement-specific checks on the resulting artifact. For things that need judgment, use the same rubric and conceal which version produced each result where practical. Inspect the evidence behind a verdict, especially when a reviewer rejects a valid alternative or misses a failure.

Keep a record per attempt with the version, case, starting revision, outcome, checks, required corrections, duration, and available usage data. Save the artifact and relevant output so an unexpected result can be investigated.

A crash or timeout still belongs in that record. Distinguish a failure of the task from an infrastructure failure or a missing evaluation. If you rerun after fixing the environment, retain the original result and explain the rerun policy. Quietly dropping the inconvenient attempts makes the comparison look better than the workflow feels.

Term2’s public benchmark results index makes two relevant cautions: missing evaluator status isn’t automatically failure, and scores from different rubrics or groups shouldn’t be compared without checking their origins. It also notes that the index doesn’t replace the underlying artifacts and logs.

The same care applies to cost. Missing usage is unknown, not zero. A token-based price estimate should be labeled as an estimate, rather than presented as an actual bill.

Let a tie be useful

Suppose both versions pass every case and neither needs correction. That result has several possible explanations. The skill may add no benefit to this workflow. The tasks may be too easy. Or the checks may miss the behavior you wanted to improve.

Before rewriting the skill, check the measurement. Would a patch that drops an account actually fail? Would an unnecessary migration be caught? Deliberately flawed sample outputs can help test the checks without requiring another round of agent runs.

Then ask whether the cases resemble your work. Keep some cases aside while revising the skill, so the final check includes work the revision wasn’t tuned against. Don’t keep adding unusual edge cases solely to manufacture a winner.

End with a decision whose scope matches the evidence:

• Keep: the change improves the outcome you care about, with acceptable cost and no material regression in the tested cases
• Revert or leave out: it adds overhead or problems without a useful benefit for this workload
• Inconclusive: the cases, checks, missing data, or variation don’t support a decision yet; name the smallest additional check that could resolve it

These are decision criteria, not results from an experiment reported here.

You don’t need a skill to win. You need to know whether it earns a place in the workflow. Sometimes the useful result is that the existing setup already does the job, and you can stop adjusting it.

Next: We’ll look at a public sidecar incident and what it reveals about keeping background work under control.