AI can run the experiments, but it cannot do the science

Key points - A Princeton-led study in 2025 found that AI agents could complete research engineering tasks but failed to produce papers good enough for a top machine-learning conference. - Researchers gave Claude Opus 4.8 six days, $3,000 in computing credits, and questions from two unpublished papers submitted to NeurIPS 2026, one of the most selective AI conferences in the world. - Both AI-written papers were rejected by the original human authors who graded them. - The agents committed too quickly to weak ideas and couldn't rethink their approach when experiments failed. - Anthropic cofounder Jack Clark called AI systems' lack of creativity a "bearish signal on short recursive self-improvement timelines."

The boldest promise in AI right now is recursive self-improvement: AI writes better AI, which writes better AI still, in an accelerating loop. Several leading labs have named it their next big milestone.

A new study from Princeton University puts a large question mark over how soon that loop might close.

What did researchers actually test?

Most tests of AI research ability give the machine a narrow, checkable task, like solving a coding puzzle or fine-tuning a small model against a fixed score. The Princeton team, led by Peter Kirgis and Sayash Kapoor, wanted to test something harder: open-ended research, the messy kind where there's no single right answer and success demands judgment, creativity, and knowing when to quit.

They developed a method they call "shadow evaluation." An AI agent is handed a real, unpublished research question, one that hasn't appeared in its training data and can't be found online, then asked to investigate it and write a publishable paper.

The questions came from two papers submitted to NeurIPS 2026. One asked whether a language model's "personas," the personality settings that shape how it behaves, can be altered by editing the model's internal weights (the billions of numbers that store everything the model has learned). The other asked how to build a detector that flags when an AI making predictions from spreadsheet data has quietly become unreliable.

Anthropix's Claude Opus 4.8, running on an open-source research framework called OpenClaw, got six days, $3,000 in API credits (pay-per-use computing access), a GPU budget for running its own experiments, a virtual computer, and full access to the internet. The original paper authors then graded what came back, exactly as they would a real conference submission.

Both papers were rejected.

Where did the AI fall short?

The agent handled the engineering well. It searched the literature, ran hundreds of experiments, and compiled results. That part impressed the human scientists.

Everything else fell apart. The agent tested hypotheses on tiny, unrealistic datasets. Its writing was hard to follow. It generated no ideas that would matter to the field.

Worse, it couldn't adapt. When an experiment failed, the agent narrowed its claims and added disclaimers rather than rethinking the approach from scratch. Feedback from helper sub-agents went ignored. Time and compute were burned unevenly, and instructions about pacing were disregarded.

"The papers were nowhere close to the mark when it came to being at the quality of a top AI conference," Kapoor said.

Kapoor traces the problem to training. Models get good at tasks that can be scored automatically, through reinforcement learning. Research judgment can't be scored that way, so models never get drilled on it.

One finding worth keeping: the agents didn't cheat. They didn't hide or misrepresent experiments, a failure mode researchers call reward hacking.

What does this mean for self-improving AI?

It means the timelines many labs are selling look optimistic. When we covered OpenAI's AI solving decade-old mathematics problems in our 11 August story, the credit dispute around those results was already a warning about how labs talk up their own work. This study adds another data point in the same direction.

In June, Anthropic published a blog post titled "When AI Builds Itself." In July, OpenAI said its model GPT-5.6 Sol helped train a smaller model, saving researchers weeks of work. Progress on narrow tasks is real.

But Anthropic cofounder Jack Clark, writing in his newsletter Import AI, said the study "rhymes with" what Anthropic found internally when it tried to automate parts of its own safety research. He described today's AI systems as "extraordinarily capable engineers" who display a "rote, formulaic thinking" that blocks genuine research.

For most people the practical reality is this: AI tools are useful for structured, checkable work. Asking them to do creative scientific thinking that produces breakthroughs is still asking too much. That gap may close, but the evidence says not yet.