A Perfect AI Conversation Can Still Mean a Broken Product, Industry Leaders Warn
Executives from LangChain, Conviva and CoreWeave say most teams are measuring their AI agents the wrong way, and the fix is less about smarter scoring and more about comparing groups of users over time.
Key points
- Harrison Chase, CEO of LangChain, said at VB Transform 2026 that many teams suffer "eval paralysis" and never ship because they chase a perfect test suite before launch.
- Hui Zhang, CTO of Conviva, showed how a single AI conversation can score well in isolation while hiding a category-wide problem visible only when comparing thousands of users against a baseline.
- LangChain fine-tuned its own small AI model to catch user-perceived errors at 10 to 100 times lower cost than using Claude Sonnet, a leading commercial AI model.
- All three executives agreed a human sign-off remains essential in fields such as legal and healthcare, even as automated checking scales up.
Picture a customer asking an AI shopping assistant for running shoes. The assistant asks a few questions, the customer buys a pair, the conversation ends. Score that interaction on its own: it looks fine.
Now look at the full picture. Across thousands of similar conversations, that assistant asked clarifying questions three times more often than normal for that shoe category, and customers completed their purchases outside the chat at five times the usual rate. Something is broken. You'd never see it by checking one conversation at a time.
That was the central argument at VB Transform 2026, first reported by VentureBeat, where leaders from three AI infrastructure companies described how enterprise teams are getting agent evaluation badly wrong. Our 14 July story "AI agents are being trusted with more decisions than companies can actually verify" found that half of enterprises had already shipped an agent that passed internal tests and then broke something for a real customer. This panel explains why.
Are teams measuring the right things?
Most teams aren't. Hui Zhang, CTO of Conviva, a streaming-analytics company, calls the better approach "contrastive analysis": comparing a group of users against a historical baseline rather than scoring individual chat logs. Single scores look fine. Population patterns reveal the actual bug. Zhang also noted the industry lacks a second data source entirely: what happens before, between and after a conversation, not just the conversation itself.
Harrison Chase, CEO of LangChain, the company behind a popular framework for building AI agents (software that can carry out multi-step tasks autonomously), named a second trap. Teams build exhaustive test suites before launch, then never ship.
"We sometimes see teams that have almost eval paralysis," Chase said. "The best teams launch and then iterate."
His framing: treat evaluation criteria like a living product specification that updates as real problems surface. "Evals are like the new PRD," he said, referring to a product requirements document, the standard spec for what an application should do.
Emmanuel Turlay, director of engineering at CoreWeave, a GPU cloud provider (GPUs are the specialised chips that handle AI's heavy computation), hit the same wall from a different angle. He chased 100 percent test coverage and still shipped bugs. His fix: set up broad always-on monitoring first, let real failures teach you which tests to write next.
Once a problem category surfaces, cost becomes the question. Turlay's rule is to start with the most capable model to prove something is solvable, then move to cheaper, smaller models for routine checks. LangChain did exactly that, fine-tuning a Qwen model (an open-source model family from Alibaba) to spot moments when users felt the agent made a mistake. It matched Claude Sonnet's accuracy at 10 to 100 times lower cost. Chase noted that many guardrails don't even need a model: simple pattern-matching code called a regex handles plenty of checks at near-zero cost.
Should humans stay in the loop?
Yes, and for reasons beyond caution. Turlay drew on his earlier work at a self-driving car company: even after compressing the retraining cycle to two weeks, a person still had to sign the model off before it went into the vehicle. The same accountability logic applies to agents handling financial or medical decisions.
Chase pushed further. The human check isn't just a safety net. "Human in the loop is really important for building trust in how these agentic systems work, and also really important for memory and learning from systems," he said. Without that feedback, the agent has nothing to improve on.
