The AI shortcut that costs companies six times more than it should

Key points

• Routing only 10 to 15 percent of cases to a large language model (LLM) cut inference costs by roughly 6x compared to sending every case through the model.
• A three-stage "cascade" pipeline handles the majority of decisions with plain rules, reserving the LLM for genuinely ambiguous cases only.
• In regulated industries, LLM-only systems often fail audits because "the model decided" is not an acceptable explanation for a specific choice.
• Asymmetric risk prompts tell the model that missing a real problem is worse than a false alarm, which changes how it behaves on borderline cases.
• Feeding human reviewer corrections back into the retrieval system is the step most teams skip, and it is why many AI pipelines plateau.

If you have ever wondered why enterprise AI products feel over-engineered compared to the chatbots you use at home, this is usually why. The stakes are different. When a hospital, a bank, or an insurer makes a wrong call, someone has to explain it to a regulator six months later. "The AI decided" is not going to cut it.

Vineet Vijay, a lead AI and machine learning engineer, published his findings via VentureBeat after spending a year building RAG (retrieval-augmented generation, a technique where an AI looks up relevant documents before answering) systems in exactly these environments. His conclusion is blunt: most teams are wasting money and quietly piling up risk by sending every decision through a large language model, the technology behind tools like ChatGPT.

So what is the smarter approach?

Stop treating the LLM as the front line. Treat it as the last resort.

Vijay calls this a cascade architecture, a three-stage pipeline that sorts decisions by how hard they actually are.

Stage one uses plain rules. Exact matches, structured data comparisons, things with clear criteria. No AI involved at all. Fully explainable, because it is a lookup. In many real systems this clears more than half the daily volume.

Stage two pulls relevant evidence for the cases that survive stage one. This is the retrieval part: digging up past reviewer decisions on similar cases, documents that explain an apparent conflict, historical precedent. Vijay is emphatic here. If you retrieve the wrong context, even the best model in the world will produce a confident, well-reasoned, wrong answer.

Stage three is where the LLM finally appears, and it only sees the residual cases that stages one and two could not resolve. In one system Vijay worked on, that was 10 to 15 percent of total volume. Result: inference costs dropped roughly 6x, and consistency on the straightforward majority became effectively perfect.

Pipeline stage	What it handles	Model involved?
Stage 1: Rules	Clear matches, structured fields	No
Stage 2: Retrieval	Ambiguous cases needing evidence	No (search only)
Stage 3: LLM	Genuinely unresolved edge cases	Yes

Does it change how you write the AI's instructions?

Yes, and this part matters more than it sounds. Most teams give the model a neutral prompt: assess whether this case should be approved or flagged. That treats both types of mistake as equally bad. They rarely are.

Missing a genuine problem can cause real harm. Wrongly flagging something clean costs a reviewer an afternoon. An asymmetric risk prompt makes that gap explicit, telling the model to treat uncertainty as a reason to escalate rather than guess. Pair that with a confidence score, and anything below your threshold goes straight to a human reviewer, whatever the model's actual classification says.

What does this mean for ordinary people?

If AI is making decisions that affect you, whether that is a loan application, an insurance claim, or a healthcare pre-approval, this architecture is what good practice looks like. It means more decisions have a clear paper trail, fewer slip through on a model's bad day, and a human reviews the genuinely difficult ones.

For teams building these tools: the question to ask before writing a single prompt is not "How do I get the model to handle this well?" It is "Which parts of this decision should never have been the model's job at all?"

Common questions

Does this only matter for big regulated companies?

Not entirely. Any team processing high volumes of similar decisions, customer support triage, content moderation, invoice matching, will see cost and consistency benefits from the same approach, even without a regulator breathing down their neck.

What happens when the LLM still gets it wrong at stage three?

That is where the feedback loop comes in. When a human reviewer overturns a model decision, that case and the correct answer should go back into the retrieval system, so future similar cases can learn from it. Skip this step and the same category of mistake repeats indefinitely.