An agent benchmark has two jobs: run the agent, then tell the truth about what happened. We obsess over the first and treat the second as plumbing. That is backwards whenever the answer is going to influence which model gets deployed.

I read the Inspect AI 0.3.284 release notes, published October 25, alongside a recent paper on agent-ranking reliability. The juxtaposition is useful. Inspect’s release repairs mundane-looking evidence failures: unexpected background-subagent errors now fail a sample; recovered samples retain their input, target, scores, output and message text; logged model inputs and outputs no longer change because an agent edits a message later. It also fixes a duplicated prompt under native compaction. These are software fixes, not new model capabilities or measured improvements in any model’s score. But without them, the evaluation record can tell the wrong story. [1]

A failed worker cannot be a successful sample

Suppose a parent agent delegates part of a task, produces a convincing final sentence, and its background worker has failed unexpectedly. If the sample is counted as success because the visible final response looks good, the benchmark has rewarded an incomplete execution. Inspect’s newly documented failure propagation addresses precisely that class of bookkeeping error. It does not guarantee that every task has an independent outcome oracle; the evaluator still needs to check the actual final state rather than accept the model’s account of it. [1]

That distinction matters outside coding. A research agent may confidently cite a document that was never retrieved. A security assistant may report a control deployed while the requested change did not take effect. A shopping agent may describe a reservation without making one. These are examples of the kind of mismatch a final-state test should catch, not claims that Inspect observed those particular incidents. If your harness grades prose rather than the effect, a polished failure can score better than an awkward success.

Now add retries. If the first attempt fails and a recovered run succeeds, what do we count? The answer depends on the question. Success on the first try, success after retries, cost to eventual success, and consistency across independent runs measure different things. A recovery system that drops the original target or scores does not merely make debugging inconvenient: it can destroy the denominator or erase evidence of an earlier failure. Inspect’s repairs to recovered sample fields and inline text are therefore relevant to the integrity of comparisons, although I have not reproduced the bugs or measured the fixes independently. [1]

More tasks do not repair the wrong inference

The ranking paper analyzes 22 agent benchmarks and distinguishes two claims. Ranking a fixed model plus scaffold is not the same as ranking an underlying model independent of its scaffold. For the individually analyzed evaluations reported in the abstract, estimated ranking reliability for fixed systems was 0.935–0.994; the corresponding underlying-model range was 0.148–0.841. Its model estimates depend on a Bayesian variance-decomposition framework, not on a universal property of all benchmarks. [2]

The authors estimate that even infinitely many similarly constructed tasks would improve model-ranking reliability by at most 0.097 in the studied setting when limited scaffold coverage dominates uncertainty. They also project that pooling diverse benchmarks can raise cross-task reliability from 0.44 to 0.75 at the same task budget. These are study-specific statistical projections, not a promise that buying a new harness will deliver those numbers. [2]

My reading: there are two independent ways to deceive yourself. Your records can be wrong about what a fixed run did, and your perfectly recorded runs can still support only a narrower comparison than the headline claims. Better logging addresses the first; varied scaffolds and tasks address the second. Neither excuses the other.

The inconvenient open issue

I would not turn this release into an advertisement for flawless logs. A still-open Inspect issue describes how rescoring a log in place with streaming and overwrite can replace a complete input log with a partial snapshot if an error or interruption occurs. The issue proposes omitting streaming or writing to a new output path as workarounds. I have not independently executed that reproduction; it is a documented report, not my test result. It illustrates the same principle as the fixes: preserve the original evidence before transforming it. [3]

The release also fixes model-log mutation when messages are edited in place. Think about the stakes: if the record of the model input changes after the response, an investigator cannot tell what was actually evaluated. A reproducible leaderboard needs immutable observations, not merely a tidy score column. Fixing the aliasing bug improves fidelity of the record; it does not establish that the grader or target is correct. [1]

What I would demand before trusting a ranking

First define the claim: this complete model–scaffold configuration on these tasks, or the model across scaffolds? Publish the scaffold, tool permissions, prompts, context handling, retry policy, evaluator version and task version. Otherwise a model name on a leaderboard is doing too much work.

Then make failure visible. Unexpected child errors should fail the sample or be explicitly reported as a separate outcome. Keep the complete initial inputs, target, tool trace, model messages, scoring rules and final-state evidence. Store an untouched original before rescoring or converting a log. If a recovery changes the task record, identify precisely which attempt contributed the score. The fact that a system can resume is not evidence that its first attempt passed.

Finally, test the evidence pipeline itself. Deliberately trigger a worker error, edit an in-memory message after logging, interrupt a recovery, and compare the resulting record with observed execution. This is a proposed validation protocol, not a claim that I ran these tests. Pair that with multiple scaffolds if the intended conclusion is about the underlying model. A row of green scores is only useful when you can reconstruct what earned them.

I have not benchmarked Inspect or any model here. The practical conclusion is narrower and more durable: version the evaluator alongside the model, preserve raw runs, and phrase the ranking at the level your design actually measures. The benchmark is part of the experimental apparatus. If it cannot accurately record failure, recovery and inputs, more samples just give you more precisely formatted confusion.

Sources

[1] https://github.com/UKGovernmentBEIS/inspect_ai/releases/tag/0.3.284 [2] https://arxiv.org/abs/2610.00651 [3] https://github.com/UKGovernmentBEIS/inspect_ai/issues/5914