AI aces trivia but fumbles riddles: the puzzle tests that expose what machines still can't do

Key points

• A Columbia University team found that in late 2024, the best AI models solved only 18% of New York Times Connections puzzles; by early 2025, some models solved them near-perfectly every time.
• A 2024 study by Google and University of Illinois Urbana-Champaign researchers found that AI models fail on slight variations of classic logic puzzles they were trained on.
• Apple researchers found that AI handles simple Tower of Hanoi puzzles (moving a stack of disks by set rules) but accuracy collapses when the number of disks reaches six or more.
• Visual and spatial reasoning remain clear weak spots for current AI, even for models that can analyse images.
• On intuition-trap questions, AI actually outperforms humans, answering carefully where we rush in with gut-feel wrong answers.

Chess. Go. Crosswords. Puzzles have been how we take the measure of machine intelligence since computing's earliest days. In 1959, IBM scientist Arthur Samuel wrote the paper that put "machine learning" into the dictionary, and his subject was a checkers algorithm.

The latest snapshot, reported by MIT Technology Review, is genuinely striking. In late 2024, a Columbia University team found the sharpest models cracked only 18% of New York Times Connections puzzles (a word-grouping game where four hidden categories link sixteen words). A few months later, some models were hitting near-perfect scores. That's a steep climb in almost no time.

But benchmarks, the standardised tests used to measure and compare AI performance, tell a richer story than a single score. They show not just how good AI is, but where it's good.

Where does AI still fall flat?

Spatial reasoning is the clearest blind spot. Mental rotation puzzles, the kind that ask you to decide whether two shapes are the same object viewed from different angles, defeat today's language models even when the model can look at images. Architects and engineers do this fluently. AI doesn't.

Memory causes a different kind of failure. Large language models (the AI technology behind tools like ChatGPT) are trained on enormous amounts of text and absorb millions of puzzle examples along the way. Feed one a classic "knights and knaves" problem (where some characters always tell the truth and others always lie) and it performs well. Tweak the wording slightly, and it often falls apart, reciting the memorised answer rather than reasoning through the change. A 2024 study from Google and the University of Illinois Urbana-Champaign confirmed exactly this pattern. Our piece on how extra memory can hurt as much as help an AI agent, published 18 August, found a related dynamic: more stored experience doesn't automatically mean better reasoning.

The ARC-AGI benchmark (a test where you study grids of coloured squares and figure out the hidden transformation rule) exposes another gap. Models do better when each grid is described as numbers rather than shown as a picture. They're processing patterns in code, not seeing the way a human does. Even when they get the right answer, researchers found they often use convoluted, patchwork rules rather than the clean visual logic a person would apply.

Scale breaks things further. Apple researchers found AI handles simple Tower of Hanoi disk-stacking puzzles and river-crossing logic problems well, right up until the number of pieces hits six. Then accuracy collapses. University of Washington, Stanford, and Allen Institute for AI researchers found the same ceiling with logic-grid puzzles (the kind where you work out who lives in which house from a list of clues).

Where does AI quietly beat us?

Intuition traps are where the machine wins. Psychologists have built question sets designed to lure humans into fast, wrong answers, phrased so the obvious-sounding reply is incorrect. AI models reason carefully. We rush in. On those tests, the machine scores higher.

AI also dominates at pure trivia recall. That vast training memory that trips it up on tweaked riddles makes it nearly unbeatable when the question is simply about a fact it stored.

The honest picture is that AI intelligence is patchy: extraordinary in some lanes, brittle in others, and moving fast enough that any score you read today may look quaint inside a year.

Common questions

Why does AI do worse when a puzzle is shown as an image rather than text?

Language models process images by converting them into numerical descriptions, not by seeing shapes the way humans do. That conversion loses spatial information, which is why grid puzzles and rotation problems are harder for the model as pictures than as strings of numbers.

Does this mean AI will never match human reasoning?

Not necessarily. These are snapshots of current models. The jump from 18% to near-perfect on Connections in under six months shows how fast things move. Spatial and visual reasoning are active research areas, and progress is expected to continue.

Should you worry that AI is getting this good at puzzles?

Not in any practical sense. These benchmarks are research tools. What they genuinely help with is understanding where AI assistants are reliable today and where a human second opinion still matters.