Apple's AI Shortcut: How a 'Draft and Check' Trick Makes Reasoning Models Twice as Fast

Key points

• Apple ML Research published a technique called Arbitrage that speeds up AI reasoning models by up to 2x without sacrificing answer quality.
• The method improves on an existing speed trick called Speculative Decoding by judging whether a draft answer is good enough, not just word-for-word correct.
• Long Chain-of-Thought reasoning, the step-by-step internal monologue that helps AI solve hard problems, is particularly expensive to run, making speed gains here unusually valuable.
• The technique targets large language models, the technology behind chatbots like ChatGPT and Claude, at the inference stage, which is when the AI actually generates an answer for a user.

Every time you ask a powerful AI chatbot a hard question, something expensive happens behind the scenes. The model thinks through the problem step by step, a process researchers call Chain-of-Thought reasoning, before giving you an answer. All that thinking burns through computing power fast.

Apple ML Research has published a paper describing a technique called Arbitrage that could make that process considerably cheaper. It's the third paper from Apple ML Research we've covered since early August, following work on AI hallucination and a new approach to text generation.

What is the problem it is solving?

The AI industry already has a speed trick called Speculative Decoding. A small, fast but less accurate model writes a rough draft of the answer, token by token, where a token is roughly a word-chunk. Then the big, accurate model checks that draft all at once. Checking in bulk is faster than generating from scratch.

There's a catch. The big model throws out tokens the moment they don't match its own prediction exactly, even when the meaning is perfectly fine. If the draft says "therefore" and the big model would have written "so", that counts as a failure. The draft gets rejected and work starts over, wasting effort.

Arbitrage fixes this by asking a smarter question. Instead of "is this word identical?" it asks "does this answer lead to a correct result?"

How does it actually work?

The system trains a small scoring model, called an advantage model, to judge whether a draft step is good enough to keep, even if the wording differs. The name Arbitrage comes from the idea of spotting value that others miss: finding draft tokens that are correct in substance even without a perfect word match.

Because fewer drafts get discarded, the fast small model contributes far more useful work. The big model spends less time regenerating steps from scratch. In testing, the approach reached speeds up to twice as fast as running the large model alone, while keeping answer quality level.

Why does this matter to ordinary users?

Faster inference, meaning faster answer generation, translates directly to shorter wait times and lower running costs for the companies building AI products. Lower costs can mean cheaper subscriptions or AI features reaching devices with less computing power.

Chain-of-Thought reasoning is what lets modern AI handle maths problems, coding and complex planning. Making it cheaper to run means those capabilities can reach more people.

Arbitrage won't change what you see on screen today. But the plumbing it improves sits under almost every serious AI product you already use.

Common questions

Does this make AI answers less reliable?

No. The advantage model is trained to accept draft steps only when they lead to correct outcomes. Answer quality in Apple's tests held steady while speed increased.

Will this appear in Apple's own products?

Apple ML Research published this as academic research. There's no product announcement yet, but techniques like this routinely move from research papers into shipping products within one to two years.