Researchers at Anthropic and OpenAI warn AI may soon improve itself faster than humans can follow
A resignation letter and a string of public warnings have put a once-theoretical AI risk into the headlines. Here is what researchers are actually worried about, in plain English.
Key points
- Evan Hubinger, an alignment lead at Anthropic, said on X that he believes there is more than a 10% chance AI could kill all humans within the next decade.
- Researchers at both Anthropic and OpenAI publicly warned this week that AI systems are already accelerating their own development faster than expected.
- Anthropic reported in August that its engineers now ship, on average, eight times as much code per quarter as they did between 2021 and 2025, partly because AI helps write the code.
- OpenAI Chief Scientist Jakub Pachocki wrote on Saturday that future systems will likely drive their own development.
- Anthropic outlined three possible futures, calling the scenario where AI remains under human control "likely", but did not rule out a path where humans play a "substantially diminished role".
The warnings started with a resignation. After a colleague quit Anthropic citing safety fears, Hubinger posted on X that he personally puts the odds of AI killing all humans within ten years at better than one in ten. More researchers at both labs followed. We first covered the resignation itself on 9 September in "An Anthropic Safety Researcher Quit. Then His Colleague Said AI Has a 1-in-10 Chance of Killing Everyone."
What is recursive self-improvement, and why does it worry researchers?
Recursive self-improvement, or RSI, is the idea that AI systems could reach a point where they help design and train the next generation of AI, which are then smarter, which then build even smarter successors, and so on. Once that loop spins fast enough, the humans who built the original systems lose the ability to steer what comes next.
AI hasn't reached that point yet, but researchers say it's already doing a version of it. Anthropic said in an August blog post that its engineers ship eight times as much code per quarter as they did between 2021 and 2025, with AI doing much of the heavy lifting. "AI is already at the level where it can introduce some new ideas," Vincent Conitzer, professor of computer science at Carnegie Mellon University, told CNBC Tech. "So it is very hard to predict at what point this process would start to drastically accelerate."
What did researchers say publicly this week?
The warnings were unusually direct. Jasmine Wang, an OpenAI researcher working on alignment (the field focused on making AI systems do what humans actually want), wrote on Wednesday that it's "hard to overstate how dangerous speeding towards RSI is." Anna Wang, who works on AGI safety at Anthropic, said there is "not yet a viable scientific plan to solve risks from recursively self-improving AI."
Pachocki wrote in a company blog post on Saturday that systems arriving in the next few years will likely "increasingly drive their own development."
| Who said it | Role | Core warning |
|---|---|---|
| Evan Hubinger | Alignment lead, Anthropic | >10% chance AI kills all humans within a decade |
| Jakub Pachocki | Chief Scientist, OpenAI | Future systems will increasingly drive their own development |
| Jasmine Wang | Alignment researcher, OpenAI | RSI is dangerous and we are heading toward it fast |
| Anna Wang | AGI safety, Anthropic | No scientific plan yet exists to manage RSI risks |
What happens next?
Anthropic laid out three scenarios. In the first, AI progress stalls and the technology spreads widely but stays limited. Anthropic called this unlikely. The second has labs making gains with humans in control. Anthropic called this one "likely." The third sees AI reach full recursive self-improvement, with humans playing a "substantially diminished role" in what gets built. What that means for the alignment problem, Anthropic admitted, is what it's "least certain about."
The honest read here: the researchers building the most powerful AI systems are genuinely unsure whether they can keep control of what they're making. That isn't hype. It doesn't mean the worst outcomes are inevitable, but the alignment problem doesn't have a solved answer yet, and the people closest to it are saying so publicly.
Common questions
Does this mean AI will definitely become dangerous?
No. Researchers describe these as real risks worth serious attention, not certainties. Anthropic itself called the scenario where humans stay in control the most likely outcome.
Should I be worried about AI tools I use today?
The concerns researchers raised are about future, far more capable systems, not the chatbots and writing assistants available now. Current tools carry their own, narrower risks such as errors and bias, separate from the RSI debate.
