AI could wipe out humanity, some researchers warn. How seriously should we take that?
A wave of alarming claims from AI insiders has pushed calls for slowing down development into the mainstream. Here is what the researchers are actually saying, what the tech companies want, and where governments stand.
Key points
- Anthropic researcher Evan Hubinger put the chance of AI killing all humans within a decade at greater than 10 percent.
- Former Anthropic employee Jacob Coxon told the BBC in 2025 that staff were "genuinely frightened" by how fast the technology is advancing.
- Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman have both called for a slowdown, though neither wants to stop model training entirely.
- President Trump dismissed the warnings during a visit to Ireland, saying the US must win the "AI race" against China.
- A cross-party group of UK MPs and peers published a report in 2025 calling for new laws to address AI's human rights risks.
A small but vocal group of researchers who build AI for a living now say the technology is advancing faster than anyone can safely manage. The concern is not science fiction. It comes from people working inside the labs.
What are researchers actually afraid of?
The core fear is losing control of AI systems before we understand them well enough to stop them causing harm. Evan Hubinger, who works in AI alignment (a field that tries to build human values into AI systems so they stay useful rather than dangerous), publicly put the odds of AI killing all humans within the next decade at above 10 percent. He stressed that risk from current tools is still low, but the trajectory worries him.
Around the same time, Jacob Coxon resigned from Anthropic and told the BBC that colleagues were "genuinely frightened" about the pace of change. Separately, Dario Amodei wrote in September 2025 that the technology had advanced "drastically faster" than expected, including in its ability to start improving itself.
That last point matters. Once an AI system can help design the next, more powerful version of itself, human engineers may struggle to keep up with what they have built. Our earlier piece on what a one-to-two-year slowdown actually solves found that nobody has yet answered the harder question: how do you keep an advanced system under control once it is already running?
Other scenarios researchers raise include highly capable AI being used to plan cyber-attacks or biological weapons, or AI agents (software tools that carry out tasks independently, without a human approving each step) acting in ways their creators did not intend.
Should ordinary people be worried right now?
Not from the apocalyptic scenarios, at least not yet. Researchers who think extinction-level warnings are overblown argue they distract from harms already unfolding: deepfakes (fake images or videos generated by AI that look real) used to create non-consensual nude images of women, and AI-generated misinformation. A UK parliamentary report published in 2025 said new laws are needed to address those everyday harms specifically. On 15 September, a UK minister told trade union leaders that the government intends to treat safety warnings as genuine obligations rather than background noise.
Both sets of concerns deserve attention.
What do the tech companies want?
Amodei and Altman are both calling for a "slowdown," which sounds dramatic but turns out to mean something specific. Neither wants to halt research. Amodei wants companies to take more time testing safety before releasing new systems and to bring in independent third-party reviewers to check their work. Altman wrote on X that development "should be slower than it otherwise could be" and that safety checks have real costs worth paying.
Critics point out that calling for regulation can also protect the companies already at the top, making it harder for smaller rivals to compete. They have form: from Elon Musk's early warnings to last week's chorus of caution, the industry's most powerful voices have repeatedly called for rules while building the very systems they say scare them.
| Voice | Position | Date |
|---|---|---|
| Dario Amodei, Anthropic CEO | Slowdown yes, halt no; wants third-party safety audits | September 2025 |
| Sam Altman, OpenAI CEO | Pacing needed; companies won't wait for governments | 2025 |
| Elon Musk, SpaceX | Has backed a slowdown in the past | Prior to 2025 |
| President Trump | Dismissed risks; says US must beat China on AI | July 2025 |
| Chinese foreign ministry | Called US threat warnings "malicious competition" | 2025 |
Trump's position is straightforward: he sees AI as a race the US must win and downplayed the warnings as negativity that "won't happen." China has stayed largely quiet on the existential debate while pushing its own development hard and calling for global governance of the technology.
The gap between "slow down to stay safe" and "slow down and lose to China" is where most of the real politics will happen from here. What to watch is whether any government moves from rhetoric to binding rules before the next capability jump arrives.
