AI Routing Sounded Like Easy Savings. The Reality Cost Twice as Much.
A team building smart AI systems thought sending simple tasks to cheaper AI models would cut costs. Then the numbers came in.
Key points
- Across 417 tasks, Claude Sonnet 4.5 cost $79 total ($0.19 per task) while GPT-4.1 cost $155 ($0.37 per task), despite GPT-4.1 having lower published pricing.
- The gap came down to caching: Sonnet reused stored context more cheaply, wiping out GPT-4.1's sticker-price advantage.
- Routing AI requests, the practice of automatically directing tasks to the most suitable AI model, is a systems problem, not a simple sorting problem.
- A lightweight optimisation-based router achieved 84% accuracy at $93 per run, cutting costs 21% and latency 9% compared to running a single top-tier model alone.
The idea sounds sensible. You have several AI models available, each with different prices and strengths. So you build a router, a piece of software that automatically directs each incoming task to the most suitable model. Easy requests go to cheaper models. Hard ones go to the expensive, powerful ones. Costs drop. Quality holds. Job done.
That is the theory. The team at Hugging Face built one of these routers in practice, and the numbers told a different story.
They ran 417 identical tasks through two models: Claude Sonnet and GPT-4.1. GPT-4.1 has lower published prices per word processed. It should have been cheaper. It cost $155. Sonnet cost $79, less than half.
The culprit was caching. When an AI processes a long task, it can save parts of that work and reuse them in later steps, the same way a browser stores a webpage so it loads faster next time. Sonnet charges less to read from that saved store. Because the tasks involved lots of repeated context, Sonnet's caching advantage wiped out GPT-4.1's lower headline rate entirely.
The pricing sheet told one story. The real bill told another.
Does this matter if you are just using an AI tool, not building one?
Yes, indirectly. Every AI product you use, from a customer service chatbot to a writing assistant, almost certainly routes your requests behind the scenes. How well that routing works affects the speed you experience, the quality of the answer you get, and ultimately the price the company charges you.
Beyond cost, the team found two more surprises.
First, task difficulty is often invisible upfront. A request that reads as simple, "summarise this contract", can trigger a chain of steps: searching documents, running compliance checks, calling external tools. You rarely know how hard a job is until the AI is already doing it.
Second, latency, meaning how long you wait for an answer, depends on far more than model size. Which hardware the model runs on, whether its cache is warm, and how busy the server is can matter more than the model itself. Adding a router adds its own delay too, especially if the router makes a fresh decision at every single step of a task.
The team's solution was to stop asking "which model is best?" and start asking "what combination of cost, speed, and accuracy works best right now?" Their router runs that optimisation in roughly 6 milliseconds and uses about 2 kilobytes of memory, small enough not to become the bottleneck.
The honest takeaway: if you are paying for an AI tool and it feels inconsistently slow or surprisingly expensive, the routing layer is worth asking about. And if you are building AI features into your own product, treat cost benchmarking as a live experiment, not a one-time spreadsheet exercise. The price on the model card is rarely the price you will actually pay.
