Your AI agent didn't hallucinate. It just did something it was never supposed to do.
A growing number of enterprise AI agents are taking real business actions without clear authority to do so. The problem isn't bad AI reasoning. It's that companies haven't defined where the AI's power stops.
Key points
- A Cloud Security Alliance survey in April 2026 found that 65% of 418 IT and security professionals had experienced an AI-agent-related incident in the prior year.
- The same survey found 82% of respondents had discovered AI agents operating in their environments that nobody knew were there.
- The World Economic Forum published a playbook in May 2026 introducing a formal "Agent Capability and Authorization Profile" to make AI agent actions auditable.
- Singapore's updated Model AI Governance Framework treats access controls, safety guardrails, and human approvals as three separate problems, not one.
- Experts say the fix is an "authority contract" for every agent, a machine-readable record of exactly what the AI is and isn't allowed to do.
The refund was the right amount. The order change matched what the customer asked for. The supplier the AI picked really was the cheapest. None of that mattered, because the AI had no business doing any of it on its own.
This is the problem quietly spreading through companies that have deployed AI agents, software that can carry out multi-step tasks on its own, inside real business systems. The agent isn't broken. The company just never told it where to stop.
Why content filters don't solve this
Most companies focused first on safety filters, tools that block an AI from saying something harmful or leaking private data. Those filters matter. But they answer a different question.
A filter asks: is this output harmful? An authority question asks: is this AI allowed to take this action at all, even if the action is perfectly safe and technically correct?
An AI agent can calculate the right refund, apply it correctly, and still exceed the dollar limit a manager would need to approve. Nobody got hurt. Nothing looked wrong. The problem only surfaces later, when finance notices the numbers.
What should companies actually do?
The answer, according to governance frameworks from Singapore's government and the World Economic Forum, is to give every production agent what some researchers call an "authority contract": a machine-enforced record of what the agent may do, what it must refer to a human, and what it must never touch regardless of how confident it is.
Every agent action should resolve to one of four outcomes:
| Outcome | What it means | Example |
|---|---|---|
| Allow | Agent acts alone | Retrieving an approved document |
| Approve | Agent prepares; human confirms | Any payment or production change |
| Recommend | Agent proposes; human decides | Legal or financial decisions |
| Deny | Agent cannot act, ever | Deleting critical data |
One point gets missed consistently. A "Deny" rule written into a plain-text instruction to the AI, the kind of note a developer types into the system setup, is not a hard boundary. It is a suggestion the AI might not follow. Technical enforcement has to sit outside the AI itself.
What does this mean for ordinary people?
If you're a customer, this is why an AI chat agent might process a refund or change an order without any human ever reviewing it. Most of the time that's fine. When it isn't, the error may not be caught quickly.
If you work at a company using AI agents, watch for actions the system takes that nobody explicitly approved. A refund that exceeded a normal limit. A supplier contract the AI accepted rather than flagged. A production system change that ran overnight with no human sign-off.
As VentureBeat has noted in its coverage of enterprise AI deployment, the most dangerous AI mistake in business is often not a hallucination. It's a technically correct action the agent had no authority to take.
What to watch for: Any AI tool at your company that can write to systems, issue credits, or confirm commitments to third parties should have a documented limit on what it can do alone. If nobody in your organisation can name that limit, it probably hasn't been set.
