The Same AI Model Scored 30% on One Benchmark and 81% on Another. IBM Says the Wrapper Is Why.

Key points

• Using the same AI model, IBM's CUGA agent framework completed 81% of test tasks versus 30% for LangChain's Deep Agents in a controlled 90-task benchmark run, a gap created entirely by the wrapper software.
• CUGA's policy system intercepted a rogue payment in a live demo, stopping an AI agent from paying $90 (including a debt in collections) and holding the transaction for human review until only the $10 legitimately owed was approved.
• IBM published the full CUGA framework as open-source code under an Apache 2.0 licence, meaning any organisation can run it on their own servers, including air-gapped systems with no internet connection.
• The governance research backing the system was presented at ACM CAIS 2026, describing five specific checkpoints where rules are enforced during every agent run.
• Where IBM's recent Granite model updates focused on what the AI knows, CUGA is about controlling what it does once deployed.

Pick the wrong frame for a picture and even the best camera produces a bad photo. That logic applies to AI agents too.

An agent framework is the software wrapper that sits around an AI model and decides how it plans, which tools it can reach, whether it checks its own work, and when it stops for a human. IBM researchers ran three of them on the same AI model across the same 90 tasks and got wildly different results.

What did the test actually find?

CUGA, IBM's own framework, completed 81% of tasks. Hermes, from Nous Research and built for personal-assistant work, came in well behind. LangChain's Deep Agents, designed for long research-style tasks, completed 30%. The model powering all three runs was identical: GLM-5.3-flash, an open-weight model whose code is publicly available. Every task came from AppWorld, a standard benchmark that mimics multi-step digital work across nine apps, covering email, payments, file storage and more.

AI2Day first covered AppWorld benchmarks on 11 August 2026 and has since reported on IBM's Granite AI work across nine stories in the past 90 days; this one lands differently because the subject is control, not capability.

The most telling number is the scenario score. A scenario in AppWorld contains three related tasks and only counts as passed if all three pass, which punishes lucky one-off completions and rewards consistency. CUGA passed 20 of 30 scenarios. Hermes passed 8. Deep Agents passed 2.

The researchers, documented in a paper on arxiv.org, are careful to note that Hermes and Deep Agents were not built for multi-app API work. Those tools aren't poor choices in general; they're poor choices for this particular job.

Why does the wrapper change the score so much?

CUGA breaks every task into steps, giving the AI model only the tools relevant to each step rather than flooding it with all 469 available options. Before any code that changes data runs, a verification step checks it. After each action, an automatic reflection stage lets the agent review what happened and correct course. Less falls through the cracks.

Those structural guardrails are also where governance enters.

Can the software actually stop a bad AI decision?

Yes, and IBM built a live demonstration of it. Send "Pay everything I owe" to an agent without policies and it paid $90, including a disputed debt in collections. The same message to an agent with CUGA's policy layer paused the payment, flagged it for human approval, and ultimately released only the $10 genuinely due.

The policy system works at five checkpoints. An Intent Guard blocks a dangerous request before the agent even starts planning. A Playbook steers the agent's reasoning by embedding a standard procedure directly into its instructions. A Tool Guide runs a hard check in code before a tool is called, returning a policy violation if the action is blocked rather than asking the model to reconsider. A Tool Approval gate pauses the entire run and waits for a human to review the proposed action. An Output Formatter shapes the final reply to an approved template.

The distinction IBM draws matters: guidance (like a system prompt) tells the model what to do. Enforcement (like a Tool Approval) stops the code from running, regardless of what the model decides. Business rules around payments or customer data belong in that second category.

CUGA is Apache 2.0 open-source, works with any service using the standard OpenAI-compatible API format, and runs on a company's own servers or a fully disconnected system. That last point is real for regulated industries where data cannot leave a controlled environment. IBM's commercial Sovereign Core product builds on CUGA for organisations that need that level of isolation.

The benchmark gap is striking, but AppWorld is a proxy for enterprise work, not a replica. What's harder to dismiss is the payment demo. That's a concrete picture of what policy-by-prompt gets wrong, and it's the part worth watching as more teams start wiring agents into systems that move real money.