Your AI Agent Scored 77%. It Only Nails the Same Task Every Time on 53% of Them

Key points

• A ReAct agent, a type of AI that reasons step-by-step before acting, scored 77.4% on average across benchmark tasks but passed all five repeated attempts on only 53.0% of those tasks.
• That 24.4-percentage-point gap between "usually works" and "always works" is what researchers call the consistency gap.
• A new tool called the Consistency Analyzer pinpoints the exact decision steps where the AI could easily go either way, with no extra full test runs needed.
• Turning those shaky moments into written guidelines for the agent cut the consistency gap roughly in half, from 24.4 points down to 12.0 points.
• Mean accuracy, the number every leaderboard reports, actually rose slightly at the same time, from 77.4% to 81.0%.

Picture an AI assistant handling your expense reports. Monday: works perfectly. Tuesday, same report: it fails. Nothing changed. That is the problem researchers at Hugging Face are now putting numbers to.

What is the consistency gap, and why does it matter?

The gap is the difference between how often an AI agent succeeds on average and how often it succeeds every single time you ask. For tasks like checking a contract or reconciling a payment, "usually" is not good enough.

Most AI benchmarks report something called Mean@k, the average pass rate across several attempts. A score of 77% sounds solid. Researchers also measured a stricter number called Pass^5, which counts only the tasks the agent completed successfully on all five tries. That figure was 53.0%. Nearly a quarter of the benchmark was tasks the AI could sometimes do and sometimes couldn't, with nothing about the question changing between runs.

This is not a problem you solve by switching to a bigger model. Capability and consistency are two separate axes, and an agent can be strong on one while failing on the other. Our coverage of AppWorld from 11 August 2026 hit the same benchmark from a different angle, and the reliability question never came up then. That gap in how the field evaluates agents is exactly what this research targets.

Why does the AI flip its answers?

Every decision an AI agent makes comes from a probability spread over possible next words or actions. Sharp distributions put most weight on one option: the same choice lands every time. Flat distributions spread weight across several close competitors, and tiny platform-level nudges, even the order in which requests arrive at the server, can tip the result either way. Chain enough flat decisions across a multi-step task and the gap between average performance and reliable performance grows fast.

Temperature, the setting that controls how random an AI's outputs are, cannot fix this. The instability lives in the probabilities themselves. The agent in this study ran at temperature 0.0, meaning no deliberate randomness was added at all.

How does the fix actually work?

The Consistency Analyzer replays each step of a recorded AI task and asks the model to produce several completions of that one moment. It flags where answers scatter widely: one recorded trace, a handful of extra calls at the wobbly steps, no access to internal model weights.

Each shaky step then becomes a written guideline fed back to the agent before it works. One real example from the benchmark: "When counting checkbox-style markers in note content, use a line-anchored regex match" (a method that checks each line individually) "rather than a plain substring count." General advice, not a one-time patch.

Results across 168 test tasks: Pass^5 climbed from 53.0% to 69.0%, and the consistency gap shrank from 24.4 points to 12.0. Nearly a third of previously unreliable tasks became tasks the agent completed correctly every time. Full methodology is in the technical report on arXiv.

A 12-point gap still remains, which matters for genuinely critical workflows. Halving the gap without hurting average accuracy is real progress, but anyone deploying agents on high-stakes tasks should treat that residual variance as a live risk, not a footnote.