Anthropic Found Five Attempts to Use Its AI for Bioweapons. That Should Alarm Everyone.
A new disclosure from the maker of the Claude chatbot shows that AI safety is only as strong as the least careful company in the room, and that the US and China may finally know it.
Key points
- Anthropic, the company behind the Claude AI chatbot, identified five separate cases in 2024 in which users attempted to use its models to help develop biological weapons.
- In at least one case, a platform redirected requests that Claude rejected to a competing AI model with weaker safety filters.
- The US and China are reportedly planning their first bilateral AI-safety talks ahead of a summit between Donald Trump and Xi Jinping.
- A researcher at Anthropic has warned that AI could pose a 10% chance of wiping out humanity, a figure the company has not publicly disputed.
- Neither country can make frontier AI, meaning the most powerful and capable AI systems, safe without the other's cooperation.
Anthropic, the San Francisco company that makes the Claude chatbot, disclosed this week that it had found and banned five accounts whose users attempted to use Claude to support biological weapons development. The company blocked the requests. But one detail buried in the disclosure is the part that sticks: in at least one case, a third-party platform simply took the rejected request and sent it to a rival AI model with fewer guardrails in place.
That is the problem in one sentence. AI safety is only as strong as the weakest system available.
Why does it matter who builds the safest AI if others don't bother?
It matters because the dangerous use goes somewhere else. A researcher at Anthropic has put the odds of AI causing human extinction at roughly 10%. That figure comes from inside one of the most safety-focused AI labs in the world, which makes it harder to dismiss than if it came from a critic outside the industry.
Think of it like food safety. If one restaurant chain enforces strict hygiene rules but the one next door ignores them, customers still get sick. The careful restaurant's standards protect nobody if people can just walk next door.
Geopolitics is making this harder. The US and China are locked in a fierce competition to build the most powerful AI systems first. Yet, as The Guardian's editorial board noted, neither country can make the most advanced AI safe on its own. Reportedly, the two governments are planning their first formal AI-safety talks before a White House summit between Donald Trump and Xi Jinping. That's genuinely new territory. Whether it produces anything useful is a separate question, and we've covered US-China AI relations only once before on this site, first in July, which tells you how recently this diplomatic thread emerged.
What does this mean for ordinary people?
For most people, none of this is immediately visible. Chatbots still answer questions about recipes and help draft emails. But the bioweapons disclosure is a concrete example of what happens when powerful AI tools are widely available and safety standards vary between companies.
Countries that deploy AI systems need a seat at the table when global rules are written. A US-China agreement, if one ever materialises, won't cover every government or every model.
| Fact | Detail |
|---|---|
| Blocked attempts | 5 cases involving biological weapons support |
| Redirect incident | At least 1 case sent to a less-restricted rival model |
| Extinction risk estimate | ~10%, per an Anthropic researcher |
| Diplomatic milestone | First US-China bilateral AI-safety talks, date unconfirmed |
| Company involved | Anthropic (maker of Claude) |
The honest read here is that voluntary safety measures from individual companies aren't a policy. They're a starting point. Five blocked attempts isn't a reassuring number; it's evidence that people are already trying.
Common questions
Is Claude, or any public AI chatbot, actually dangerous to use?
For everyday tasks, no. The safety risk described here involves deliberate, organised attempts to extract harmful information, not normal users asking normal questions. Anthropic's filters caught the attempts described.
What are AI-safety talks between the US and China meant to achieve?
The goal is to agree on shared limits: what AI should never be allowed to do, even when the two countries disagree on almost everything else. Think of it as the nuclear non-proliferation treaty model applied to software.
