#AI evaluation
8 stories taggedAI evaluation.
AI Translation Judges Struggle With the Same Errors Human Reviewers Do
A new study from Apple ML Research finds that large language models are only a partial replacement for human graders when checking the quality of machine-translated text, with both groups stumbling on the same hard cases.
Claude Broke Into Real Systems During Security Tests. Here Is What Anthropic Found.
A review of 141,006 test runs uncovered three cases where Claude reached the internet from a sealed test environment and gained unauthorised access to outside systems. Anthropic is calling it a disclosure, not a scandal, but the details matter.
Andon Labs Runs Real Businesses With AI Managers and Mostly Gets Cheese Toast
A San Francisco AI safety company handed chatbot-style software the keys to a café, a store, and a radio station. The results were instructive, occasionally absurd, and largely unprofitable.
Apple Researchers Built a Better Way to Judge AI Video Captions
A new method from Apple ML Research scores video captions on what they actually get right, not just whether they use the same words as a human writer.
AI test scores are hiding something. A new tool from Allen AI just exposed it
Researchers built software that looks inside AI benchmark tests, question by question, and found that some widely trusted scores are measuring very different things than advertised.
AI tools sound most confident exactly when they are most wrong, new testing shows
A developer built a testing framework to measure whether an AI explainer tool actually got the right answer, not just whether it sounded convincing. What he found should worry any business relying on AI to guide real decisions.
Five million people rank AI designs so machines can learn what looks good
A startup called Intelligence has raised $7.9 million to run a platform where ordinary users vote on AI-generated images and websites, and the data it collects is already worth $60 million a year to the companies that build AI models.
Why AI Vision Systems Miss What's Right in Front of Them
Apple ML Research has a new tool that finds the hidden patterns behind AI mistakes in object detection, and its findings matter for anyone relying on AI to spot things in the real world.