Picking the Right AI Model Is the Easy Part
The engineering layer around an AI model now sets enterprise cost and reliability more than the model itself does. Here's what that shift demands.
Key points
- The serving harness, the software layer that queues requests, allocates memory and schedules compute, is now the primary driver of enterprise AI cost and reliability.
- GPU memory limits, not raw computing power, are the most common bottleneck that causes AI systems to slow down or fail under real user traffic.
- Enterprises that build their AI infrastructure around specific workload requirements can swap in a newer model later without rebuilding the whole system.
- A proof-of-concept AI that looks brilliant in a demo can still fail in production if the surrounding system isn't engineered to handle real-world load.
For three years, enterprise AI conversations have centred on models: which one scores highest, which one reasons best, which one handles the longest documents. That framing has quietly become the wrong question.
The question companies now face is harder: can you actually deliver that model's output to real users, at scale, without the economics blowing up?
According to EXL's engineering research, published through MIT Technology Review, the answer increasingly lives in what engineers call the serving harness. It's the scaffolding around the model: the software layer that decides how requests are queued, how memory is allocated, how many users can be served at once, and how compute resources are scheduled. None of that is glamorous. All of it is decisive.
Why does the harness matter more than the model?
In production, a model that gives marginally better answers isn't automatically the right choice if it costs twice as much to run or can't respond fast enough for the application.
A customer-facing chat assistant needs to respond in under two seconds. An overnight document-processing pipeline can take its time. Those two workloads could run on the same foundation model, a large AI system trained on broad data, yet require completely different infrastructure. Our story on who actually controls AI costs, published on 20 October, found the same pattern: the bill is set by deployment decisions, not by which model sits underneath.
What is the GPU memory problem, exactly?
GPU memory limits are the most common chokepoint, and they trip up even well-resourced engineering teams.
During inference, the process of generating a response, the model's core weights stay fixed in memory. But a structure called the KV cache (short for key-value cache), which stores context from the ongoing conversation, grows with every active request. Under real traffic, a system can exhaust usable GPU memory even when raw computing power is still sitting idle. Adding more GPUs without fixing that specific bottleneck just makes an inefficient system cost more.
Techniques like quantisation (shrinking model weights to use less memory without much accuracy loss) or speculative decoding (a method that drafts several tokens at once to speed up responses) only help once the primary constraint is identified. Apply them in the wrong order and the gains are marginal at best. EXL's research is direct on this: measure first, find the dominant constraint, address it, then check whether a new bottleneck has appeared.
What should an ordinary business take from this?
If your organisation is moving an AI project from pilot to production, the engineering choices made in the next few months will set your costs and your ceiling for years.
Build the surrounding system around the job you need done, not around today's leading model. The model will change. Hardware will improve. A well-designed serving layer can absorb those changes as configuration decisions rather than full rebuilds. That's the practical definition of not painting yourself into a corner.
For anyone running AI on consumer hardware, the same logic applies at smaller scale: our 20 October story on PyTorch 2.15 showed how framework-level engineering cut memory use for video generation from 270 gigabytes to 8, pushing some language models past 120 tokens per second on Apple silicon. The model didn't change. The layer around it did.
What this beat keeps showing is that model benchmarks are a distraction from the more durable question: who on your team owns the production infrastructure, and do they have the authority to make it the first investment rather than the last?
