The most flattering number in agent AI is pass@1: give a model a task once, count success, publish the percentage.

It is a useful number. It is also radically insufficient for deciding whether an agent gets permission to touch anything consequential.

A new Princeton-led study makes the uncomfortable case cleanly. It evaluated 15 models on agent benchmarks and separated reliability into consistency, robustness, predictability, and safety. Capability climbed. Reliability improved only modestly. Newer models were more likely to solve a task, but they were not reliably more repeatable, more stable under small changes, or better at knowing when they would fail.

That is not an academic distinction. It is the difference between a clever demo and an employee with a keycard.

One Good Run Is Cheap

Pass@1 answers a narrow question: could this system complete this task in this particular attempt?

That is exactly the question vendors want answered, because it makes the chart go up. It is not the question an operator needs answered.

The operator needs to know:

  • Does it still work after the request is phrased differently?
  • Does it still work on Tuesday?
  • Does it stay within its authority when it is wrong?
  • Does it stop when the evidence no longer supports its plan?
  • Does it cost roughly the same amount each time?

Those properties do not automatically arrive when a model becomes more capable. The Princeton study found that on the more open-ended GAIA benchmark, reliability gains were especially thin even as models advanced. More intelligence produced better shots on goal. It did not produce a dependable game.

That should change what we buy, benchmark, and deploy.

Repetition Is Where the Invoice Appears

The long-horizon reliability study is more brutal. Its authors ran 10 models through 23,392 episodes over tasks of different durations. The headline is not that agents get worse over longer work. Everyone knew that in their bones.

The headline is how badly our usual metric hides it.

On software-engineering tasks, their Graceful Degradation Score fell from 0.90 on short tasks to 0.44 on very long tasks. Document-processing tasks were almost flat, from 0.74 to 0.71. In other words: there is no single answer to “is this agent reliable?” Reliability depends on the job, the horizon, and the shape of the environment.

The study also reports a nasty inversion. The strongest models can show the highest long-horizon meltdown rates, up to 19%, because they attempt ambitious multi-step strategies. Weak models often fail early and boringly. Strong models can make convincing progress for forty minutes before entering a loop, contradicting their earlier work, or inventing tool results.

That is a more expensive failure mode. It arrives after the system has accumulated context, authority, and user trust.

A one-shot benchmark will celebrate the ambitious model. A production system has to pay for the 19%.

Stop Asking Whether the Model Is “Reliable”

“Reliable” is too blunt to be useful. Treat it as a property of a complete workflow instead.

A useful agent reliability card should include at least four numbers:

  1. Repeated-run success. Not one attempt. Run the same task enough times to expose variance.
  2. Perturbation stability. Change harmless wording, ordering, or irrelevant context. Does the result survive?
  3. Bounded failure. When the agent is wrong, does it fail safely, ask for confirmation, or make the situation worse?
  4. Recovery cost. How much human time, compute, and cleanup does a failed run consume?

The fourth number is the one benchmark culture keeps trying to lose in the sofa cushions. A failed draft is cheap. A failed action after ten tool calls, three edits, and a downstream handoff is not. If the agent is allowed to act, recovery cost is part of accuracy.

Memory Does Not Magically Fix This

The long-horizon study tested a memory-augmented scaffold alongside a standard ReAct loop. The memory version did not rescue performance. It was neutral or harmful for long-horizon graceful degradation across all ten models.

That result should not be surprising. Memory is more state. More state means more old assumptions, more chances to retrieve the wrong note, and more material for a model to rationalise around.

The answer is not “never use memory.” It is to stop treating memory as a reliability upgrade by default. Memory needs provenance, expiry, conflict handling, and checks at the moment it influences an action. Otherwise it is just a very efficient way to preserve yesterday’s mistake.

Build the Evidence Loop

The fix is not to wait for a mystical reliable model. The practical move is to make agents easy to prove wrong before they can make damage expensive.

That means:

  • narrow permissions by default;
  • split irreversible actions from research and drafting;
  • make expected end states machine-checkable;
  • preserve traces that a human can inspect without reading a novel;
  • run high-value tasks repeatedly before granting more authority;
  • measure recovery time, not just success.

The real product is not the model plus a tool belt. It is the evidence loop around the model: tests, constraints, independent checks, and a route back when the first plan was wrong.

Pass@1 is not fraudulent. It is just the beginning of the bill.

The dangerous mistake is treating it as the receipt.