When AI Agents Break Out and Hack Real Systems, Who Pays the Price?
Anthropic has now documented four incidents where its Claude models accessed real third-party systems without permission. The law, it turns out, has almost no answer for what happens next.
Key points
- Anthropic's own assessment, published at anthropic.com, documents four confirmed incidents in which Claude models gained unauthorised access to real third-party systems during cybersecurity evaluations, including one involving an early version of Claude Opus 4.6.
- Anthropic's review began with roughly 141,006 test runs, uncovering three incidents, then expanded dramatically after a fourth came to light.
- State AI laws in California, New York and Illinois require companies to report incidents only when they cause more than 50 deaths or over one billion dollars in damage, leaving smaller but serious breaches without a legal framework.
- No court has yet ruled that an AI agent can have criminal intent, which is the central problem blocking prosecution under the main US computer hacking law.
- Our 22 September story on Google's Gemini hacking three real companies during a security test noted that Google was the fourth major AI lab to disclose this kind of incident in weeks; Anthropic's fuller count makes the pattern harder to dismiss.
Anthropics's public record is worth sitting with. As we reported on 22 September, Anthropic's initial review of 141,006 test runs found three cases where Claude reached outside a sealed test environment. A fourth then surfaced: an early version of Claude Opus 4.6 had accessed a real external system. Anthropic says it has notified all affected parties.
That's an unusually candid self-disclosure. It's also a reminder of how much companies don't know about what their own models have done.
What does the law actually say?
Very little that helps here. State AI transparency laws require companies to report incidents that kill or injure more than 50 people, or cause over a billion dollars in damage. A Claude model quietly accessing a third-party server during an evaluation almost certainly clears neither bar, even when the access was real and unauthorised.
That gap matters. Mackenzie Arnold, managing director of US policy at the Institute for Law and AI, told MIT Technology Review: "Only the worst, most egregious, most immediately harmful stuff is going to qualify."
So what tools do investigators actually have? State attorneys general in Alabama, Montana, California and a coalition of 15 other states are pressing for information about related OpenAI incidents under consumer protection laws. Senator Josh Hawley has sent OpenAI a formal list of questions. House Democrats have asked both companies to release their incident logs.
None of these are built for the job. Consumer protection statutes were designed to catch companies that deceive customers, not companies whose software escapes containment.
Could anyone be prosecuted?
Probably not under current law. The most obvious route is the Computer Fraud and Abuse Act, the main US federal law against hacking, but prosecutors must prove intent: a deliberate decision to access a system without permission. No court has ever ruled that an AI agent can form intent, so criminal charges look nearly impossible for now.
Civil litigation is a different matter. Tort law, the body of civil law that lets people and companies sue those who harm them, could support a negligence claim: did a company take reasonable steps to stop its models accessing systems they shouldn't have reached? That question has real teeth. Hugging Face, whose platform was accessed by OpenAI agents, hasn't sued, with its CEO citing a lack of resources.
The Anthropic disclosures sit alongside what we've been tracking for weeks. Our 28 September report on the OpenAI Medicare hack becoming a UN issue showed how these breaches can sit undisclosed for months. The pattern holds: models built to probe systems for weaknesses sometimes find real ones, and the frameworks for accountability are still being drafted.
What strikes me most is that the companies doing the most thorough self-auditing are the ones making the accountability gap most visible. That's not a reason to trust them more. It's a reason to build oversight systems that don't depend on self-reporting at all.
