Apple Researchers Built a Tool That Writes Its Own AI Tests

Key points

• Agent Seer, developed by Apple ML Research, automatically generates evaluation scenarios for AI agents by reading tool specifications rather than writing tests by hand.
• Current AI agent benchmarks go out of date every time an API or software tool changes its behaviour.
• The system works without running any live tools, keeping both cost and complexity down.
• Agent Seer targets a gap that grows as AI agents, software that can carry out multi-step tasks on its own, become more common in real workplaces.

Testing an AI agent is harder than it sounds.

An AI agent, software that carries out multi-step tasks on its own, doesn't just answer a question. It calls external tools, checks the results, and then tries again across several conversation turns. To know whether it's doing that well, you need realistic scenarios that reflect how real users actually combine those tools.

Until now, building those scenarios meant hiring domain experts to write them by hand. That's slow, it breaks the moment an API, the connection point between software services, updates its behaviour, and it doesn't scale.

What does Agent Seer actually do?

It reads the tool specification itself and writes the test scenarios from that.

Every software tool comes with a specification: a function name, a plain-English description of what the tool does, and a typed schema listing the parameters it accepts. Agent Seer, detailed by Apple ML Research, treats those specifications as a source of meaning. The system extracts enough information from them to generate realistic multi-turn conversation scenarios automatically, without ever running the tool live.

Think of it as a teacher who writes a credible exam question simply by reading the syllabus, without sitting through a single lecture.

That matters for two reasons. First, it scales: a team can cover hundreds of tools without writing one test by hand. Second, when an API changes its specification, Agent Seer can regenerate the scenarios to match, keeping the benchmark current.

Why does this matter for ordinary people?

Most people won't interact with Agent Seer directly. But the quality of AI agent testing shapes how reliably those agents behave in products real people already use.

If you use an AI assistant to book a meeting or pull data from a business system, that assistant is an agent calling external tools. Poor testing means those agents fail quietly, booking the wrong time or returning stale data without flagging an error. Better automated evaluation is one way the industry can catch those failures before they reach users.

This is also a live concern. Our coverage of AI benchmark failures, going back to the first story in this series on 16 July 2026, has tracked eight benchmark-related stories in the last 90 days, including the finding that large language models update their beliefs inconsistently. Automated scenario generation won't fix reasoning problems, but it removes one excuse for skipping evaluation altogether.

What happens next?

Agent Seer addresses a structural problem: the more tools an AI agent can use, the harder it becomes to keep evaluation benchmarks current. Tool specifications, it turns out, already contain enough information to close that gap without manual effort.

Adoption is the open question. Benchmark frameworks only improve reliability if developers actually use them, and whether Agent Seer becomes a standard part of the agent-development pipeline depends on how cleanly it fits the tool ecosystems practitioners already work with.

Common questions

Do I need to understand AI development to care about this?

No. This is infrastructure work that improves AI products behind the scenes. Better testing means the AI tools you already use are less likely to behave unexpectedly.

Does Agent Seer test tools in a live environment?

No. It generates test scenarios by reading tool specifications only, without executing any real tools. That's part of what makes it practical to scale.

Is Agent Seer available to developers now?

The research describes the system and its approach, but no public release date or open-source repository was announced in the published work.