#benchmarks
14 stories taggedbenchmarks.
Mistral's New Forecasting Model Tops Three Major Benchmarks at a Fraction of the Size
Timestral, a family of time-series models from Mistral AI, beats larger rivals on every major public forecasting benchmark and runs on an ordinary office server without specialist chips.
The Agent Said It Was Done. The Database Knew Better.
A new benchmark from Microsoft and Hugging Face runs AI agents through real business tasks 20 times each, then checks the records they leave behind. Most models fail far more often than a single test suggests.
The Best AI Agents Still Fail at Most Real Office Work. ServiceNow Built a Tool to Fix That.
ServiceNow's AutoSynthData spots where an AI agent breaks down on the job, then automatically generates practice tasks to fix the gap. The backdrop: the strongest model tested cleared barely a third of realistic enterprise tasks.
H Company's Holo4 Handles Every Interface a Computer Offers, in One Model
Holo4 can click menus, write code, and call APIs within a single task. For businesses weighing agent costs, that breadth at a smaller scale is the real story.
UK Government AI Lab Publishes Benchmark Results Anyone Can Check
The UK AI Security Institute is releasing verified test results for six frontier AI models under a shared open standard, so researchers worldwide can see exactly how scores were produced and repeat the work themselves.
Your AI Agent Scored 77%. It Only Nails the Same Task Every Time on 53% of Them
A new study puts hard numbers on the gap between 'usually works' and 'always works', then builds a tool to close it.
Apple Researchers Built a Better Way to Judge AI Video Captions
A new method from Apple ML Research scores video captions on what they actually get right, not just whether they use the same words as a human writer.
AI test scores are hiding something. A new tool from Allen AI just exposed it
Researchers built software that looks inside AI benchmark tests, question by question, and found that some widely trusted scores are measuring very different things than advertised.
Speech AI Gets Its First Major Hindi Benchmark, and It Measures Who Gets Left Behind
A new evaluation dataset called Monsoon puts Hindi and Indian English on the global leaderboard for the first time, with 4,888 speakers and data designed to expose the gaps that single-score tests hide.
Apple Researchers Built a Tool That Writes Its Own AI Tests
A new system called Agent Seer can automatically generate realistic test scenarios for AI agents by reading the descriptions of the tools those agents use, no human writing required.
Google DeepMind puts its AI in a cryptographic sealed box for testing
A first-of-its-kind double-blind evaluation aims to stop AI models from secretly studying the exam before test day.
Five million people rank AI designs so machines can learn what looks good
A startup called Intelligence has raised $7.9 million to run a platform where ordinary users vote on AI-generated images and websites, and the data it collects is already worth $60 million a year to the companies that build AI models.
Researchers Build a Test to See If AI Can Actually Summarise a 16-Minute Video
A new benchmark called LVSum reveals how badly today's AI video tools lose track of when things happen in long recordings.
An AI Taught Itself to Write Better Code, No Teacher Required
A surprisingly simple trick let a leading AI model lift its coding score by nearly 13 points, just by studying its own answers.