Your AI agent didn't hallucinate. It just did something it was never supposed to do.
Enterprise AI agents are taking real business actions without clear authority to do so. The problem isn't bad AI reasoning. It's that companies haven't defined where the AI's power stops.
Key points
- A Cloud Security Alliance survey in April 2026 found that 65% of 418 IT and security professionals had experienced an AI-agent-related incident in the prior year.
- The same survey found 82% of respondents had discovered AI agents operating in their environments that nobody knew were there.
- The World Economic Forum published a playbook in May 2026 introducing a formal "Agent Capability and Authorization Profile" to make AI agent actions auditable.
- Singapore's updated Model AI Governance Framework treats access controls, behavioral guardrails, and human approvals as three separate controls, not one.
- The fix most governance frameworks recommend is an "authority contract" for every agent, a machine-enforced record of exactly what the AI is and isn't allowed to do.
The refund was the right amount. What the customer asked for matched the order change exactly. The supplier really was the cheapest option. None of that mattered, because the AI had no business acting alone on any of it.
This is the problem quietly spreading through companies that have deployed AI agents, software that can carry out multi-step tasks inside real business systems without human prompting at each step. The agent isn't broken. Nobody told it where to stop.
Why content filters don't solve this
Most companies focused first on safety filters, tools that block an AI from saying something harmful or leaking private data. Those filters matter, but they answer a different question than authority does.
A filter asks whether an output is harmful. Authority asks whether the AI was allowed to take a specific action at all, even when that action is perfectly safe and technically correct. An AI agent can calculate the right refund, apply it without error, and still exceed the dollar limit a manager would need to approve. Nothing looks wrong. The problem only surfaces when finance notices the numbers.
Our 8 August story on AI kill switches showed companies reaching for emergency override controls after exactly this kind of runaway agent behaviour. Authority contracts are the upstream answer to the same problem.
What should companies actually do?
Governance frameworks from Singapore's government and the World Economic Forum both point to the same structure: give every production agent what some researchers call an "authority contract", a machine-enforced record of what the agent may do, what it must refer to a human, and what it must never touch regardless of confidence level.
Every agent action should resolve to one of four outcomes:
| Outcome | What it means | Example |
|---|---|---|
| Allow | Agent acts alone | Retrieving an approved document |
| Approve | Agent prepares; human confirms | Any payment or production change |
| Recommend | Agent proposes; human decides | Legal or financial decisions |
| Deny | Agent cannot act, ever | Deleting critical data |
One point gets missed consistently. A "Deny" rule written into a plain-text instruction, the kind of note a developer types into the system setup, is not a hard boundary. It is a suggestion the AI might not follow. Enforcement has to sit outside the AI itself, in a policy layer that evaluates every proposed action at runtime before it runs.
Should you worry?
If you're a customer, this is why an AI chat agent might process a refund or change an order without any human reviewing it. Most of the time that's fine. When it isn't, the error may not surface quickly.
If you work at a company using AI agents, watch for actions the system takes that nobody explicitly approved: a refund that exceeded a normal limit, a supplier contract the AI accepted rather than flagged, a production change that ran overnight with no sign-off.
Any AI tool that can write to systems or confirm commitments to third parties should have a documented ceiling on what it can do alone. If nobody in your organisation can name that ceiling, it probably hasn't been set. That gap, not a hallucination, is the likeliest source of the next quietly expensive mistake.
The honest judgement from this beat: authority gaps are harder to audit than bad outputs, they leave almost no trace until something downstream breaks, and most companies are still treating them as a future problem.
