AI aces trivia but fumbles riddles: the puzzle tests that expose what machines still can't do

Key points

• In late 2024, the best AI models solved only 18% of New York Times Connections puzzles; by early 2025, some models solved them near-perfectly every time.
• A 2024 study by Google and University of Illinois researchers found that AI models fail on slight variations of classic logic puzzles they were trained on.
• Apple researchers found that AI handles simple Tower of Hanoi puzzles (moving a stack of disks by set rules) but breaks down when the number of disks reaches six or more.
• Visual and spatial reasoning remain clear weak spots for current AI, even for models that can analyse images.
• On intuition-trap questions, AI actually outperforms humans, answering carefully where we rush in with gut-feel wrong answers.

Chess. Go. Crosswords. Since the earliest days of computing, games and puzzles have been how we take the measure of machine intelligence. In 1959, IBM scientist Arthur Samuel wrote the paper that put the phrase "machine learning" into the dictionary, and the subject of his research was a checkers algorithm. The tradition never stopped.

The latest snapshot, reported by MIT Technology Review, is fascinating. Judged purely on puzzle-solving, AI has sprinted forward. In late 2024, the sharpest models cracked only 18% of New York Times Connections puzzles (a word-grouping game where four hidden categories link sixteen words). A few months later, some models were hitting near-perfect scores. That is a genuinely steep climb.

But benchmarks, standardised tests used to measure and compare AI performance, tell a richer story than a single score. They show not just how good AI is, but where it is good.

Where does AI still fall flat?

Spatial reasoning is the clearest blind spot right now. Mental rotation puzzles, the kind that ask you to decide whether two shapes are the same object viewed from different angles, defeat today's language models even when the model can look at images. Architects and engineers do this fluently. AI does not.

Memory causes a different kind of failure. Large language models, the AI technology behind tools like ChatGPT, are trained on enormous amounts of text and absorb millions of puzzle examples along the way. Feed one a classic "knights and knaves" problem (where some characters always tell the truth and others always lie) and it performs well. Tweak the wording slightly, and it often falls apart, reciting the memorised answer rather than actually reasoning through the change. A 2024 study from Google and the University of Illinois Urbana-Champaign confirmed exactly this pattern.

The ARC-AGI benchmark (a test where you study grids of coloured squares and figure out the hidden transformation rule) exposes another gap. Models do better when the grid is described as numbers rather than shown as a picture, which tells you something important: they are processing patterns in code, not "seeing" the way a human does. Even when they get the right answer, researchers found they often use convoluted, patchwork rules rather than the clean visual logic a person would apply.

Scale breaks things further. Apple researchers found AI handles simple Tower of Hanoi disk-stacking puzzles and river-crossing logic problems perfectly well, right up until the number of pieces hits six. Then accuracy collapses. University of Washington and Stanford researchers found the same ceiling with logic-grid puzzles (the kind where you work out who lives in which house from a list of clues).

Where does AI quietly beat us?

Intuition traps are where the machine wins. Psychologists have built question sets designed to lure humans into fast, wrong answers, questions phrased so the obvious-sounding reply is incorrect. AI models slow down and reason carefully. We rush in. On those tests, the machine scores higher.

AI also obliterates humans at pure trivia recall. That same vast training memory that trips it up on tweaked riddles makes it almost unbeatable when the question is simply "what is the answer to a fact you stored".

The honest picture is that AI intelligence is patchy. Extraordinary in some directions, brittle in others, and improving fast enough that any score you read today may look quaint inside a year.

Common questions

Why does AI do worse when a puzzle is shown as an image rather than text?

Language models process images by converting them into numerical descriptions, not by "seeing" shapes the way humans do. That conversion loses spatial information, which is why grid puzzles and rotation problems are harder for the model as pictures than as strings of numbers.

Does this mean AI will never match human reasoning?

Not necessarily. These are snapshots of current models. The jump from 18% to near-perfect on Connections in under six months shows how fast things move. Spatial and visual reasoning are active research areas, and progress is expected to continue.

Should I be worried that AI is getting this good at puzzles?

Not in any practical sense. These benchmarks are research tools, not real-world threats. What they genuinely help with is understanding where AI assistants are reliable today and where a human second opinion still matters.