A Perfect AI Conversation Can Still Mean a Broken Product, Industry Leaders Warn

Key points

• Harrison Chase, CEO of LangChain, said at VB Transform 2026 that many teams suffer "eval paralysis" and never ship because they chase a perfect test suite before launch.
• Hui Zhang, CTO of Conviva, demonstrated how a single AI conversation can score well in isolation while hiding a category-wide problem visible only when comparing thousands of users against a baseline.
• LangChain fine-tuned its own small AI model to catch user-perceived errors at 10 to 100 times lower cost than using Anthropic's Claude Sonnet, a leading commercial AI model.
• All three executives agreed a human sign-off remains essential in regulated fields like legal, finance and healthcare, even as automated checking scales up.

Picture a customer asking an AI shopping assistant for running shoes before a half marathon. The assistant asks a few questions, the customer buys a pair, and the conversation ends. Score that interaction on its own: it looks fine.

Now look at the full picture. Across thousands of similar conversations, that assistant asked clarifying questions three times more often than normal for that shoe category. Customers completed their purchases outside the chat at five times the usual rate. Something is clearly broken. But you would never see it by checking one conversation at a time.

That was the central argument made at VB Transform 2026, first reported by VentureBeat, where leaders from three AI infrastructure companies described how enterprise teams are getting agent evaluation badly wrong.

Are teams measuring the right things?

Most teams are not. Hui Zhang, CTO of Conviva, a streaming-analytics company, calls the better approach "contrastive analysis": comparing a group of users against a historical baseline rather than scoring individual chat logs. Single scores look fine. Population patterns reveal the actual bug.

Harrison Chase, CEO of LangChain, the company behind a popular framework for building AI agents (software that can carry out multi-step tasks on its own), added a second trap. Teams build exhaustive test suites before launch, then never ship.

"We sometimes see teams that have almost eval paralysis," Chase said. "The best teams launch and then iterate."

His framing: treat evaluation criteria like a living product specification that updates as real problems surface, not a checklist you finish once.

Emmanuel Turlay, director of engineering at CoreWeave, a GPU cloud provider (GPU refers to the specialised chips that do the heavy number-crunching AI needs), hit the same wall from the engineering side. He chased 100 percent test coverage and still shipped bugs. His fix: set up broad always-on monitoring first, let real failures teach you which tests to write next.

Once a problem category surfaces, you also have a cost question. Turlay's rule is to start with the most capable AI model to prove something is solvable, then move to cheaper, smaller models for routine checks. LangChain did exactly that. It fine-tuned a Qwen model (an open-source model family from Alibaba) to spot moments when users felt the agent made a mistake. The result matched Claude Sonnet's accuracy at a fraction of the price. Chase noted some guardrails do not even need a model: simple pattern-matching code, called a regex, handles many checks at near-zero cost.

The human question runs underneath all of this. Turlay drew on his previous work at a self-driving car company: even after compressing the testing cycle to two weeks, a person still had to sign the model off before it went into the vehicle. The same logic applies to AI agents handling medical advice, financial decisions or legal documents.

Chase went further, arguing the human check is not just a safety net. It is how the system learns. Without human feedback in the loop, the agent has nothing to improve on.