AI Systems Fail a Basic Test of Rational Thinking, Researchers Find

Key points

• Researchers found that large language models, the technology behind tools like ChatGPT and Claude, don't update their beliefs about uncertain information in a consistent or mathematically rational way.
• The study introduces a new measurement called the "information processing gap," which tracks how far an AI's reasoning drifts from the statistically correct answer.
• The findings matter most in high-stakes fields such as medicine and law, where AI tools are already being used to help make decisions.
• The research was conducted by Apple ML Research.

What is the problem, exactly?

Statisticians have a formal, mathematically proven method for updating beliefs correctly, called Bayes' theorem, named after the 18th-century minister Thomas Bayes. It's a recipe: take what you believed before, weigh fresh evidence, arrive at a new level of confidence that's neither inflated nor deflated.

Apple ML Research asked whether large language models, the technology powering today's AI chatbots, follow that recipe. They don't. Not consistently, and not reliably.

What did researchers actually measure?

The team treated each AI model as an information-processing system, much as you'd treat a thermostat or a calculator, and measured how much its belief updates deviated from the mathematically correct answer. They named that gap the "information processing gap."

Here's how it works in practice: show the AI a piece of evidence, record its confidence before and after, then compare that shift to the shift a mathematically perfect reasoner would have made. The bigger the gap, the worse the reasoning.

The gaps were real and measurable. More troubling, the same model could handle near-identical problems very differently depending on how a question was worded or how evidence was presented. That inconsistency is the internal contradiction at the heart of the paper.

Should this worry you if you use AI tools at work?

It depends on the task. Drafting an email or summarising a document? Inconsistent probabilistic reasoning probably doesn't matter much. But a hospital weighing diagnostic evidence with an AI tool, or a legal team using one to assess case risk, faces a different situation entirely.

We've covered AI's role in clinical settings six times since July, starting on 24 July 2026, and the question of whether these systems reason reliably has been present in almost every piece. This study gives that concern a precise, measurable shape for the first time.

The research doesn't say AI tools are useless in these fields. It says their limitations are poorly understood, and that overconfidence in their outputs is a genuine risk.

The practical message is this: treat AI-generated assessments in high-stakes situations as one input among several, not as a final answer. A clinician or a legal professional should check AI reasoning against their own judgement, especially when evidence is mixed or evolving.

What happens next?

Quantifying the information processing gap gives researchers a concrete tool to compare models and, eventually, to pressure-test improvements. That's genuinely useful progress, even if the headline finding is a flaw rather than a feature.

Expect this measurement approach to appear in future model evaluations and, potentially, in regulatory discussions about which AI systems are suitable for high-stakes deployment. Apple ML Research has published several papers this summer probing where its models break down; our 19 August story on the hidden maths problem slowing AI search agents found a similar pattern of hard limits beneath a capable surface.

Common questions

What does "Bayesian" mean in plain English?

Bayesian reasoning is the mathematically correct way to update your beliefs when you get new evidence. Bayes' theorem tells you exactly how confident you should be after each new piece of information arrives, no more, no less.

Does this mean AI should not be used in medicine or law?

Not necessarily. It means AI tools used in those fields need much more careful testing and human oversight. The study identifies a specific flaw; it doesn't condemn every application.

Which AI models were tested?

The paper doesn't name a narrow set of winners or losers. The findings apply broadly across the category of large language models currently in use.