AI Translation Judges Struggle With the Same Errors Human Reviewers Do

Key points

• When grading machine-translated text, large language models agreed with human expert reviewers only modestly, performing no better than humans agree with each other on the hardest annotation tasks.
• The study tested models under two standard quality-checking frameworks, MQM and ESA, across multiple language pairs and error types, finding neither method made AI graders reliably accurate.
• Both AI and human graders struggled most with fine-grained error labelling, where annotators must identify exactly what kind of mistake a translation contains, not just whether it is wrong.
• AI2Day has tracked this territory under "AI evaluation" since 27 July 2026, and the pattern of AI underperforming at subtle human-judgement tasks keeps recurring.

Machine translation, the technology that automatically converts text from one language to another, is now good enough that checking its quality has become a serious bottleneck. Human expert reviewers are expensive and slow. Researchers have started asking whether large language models, the technology behind chatbots like ChatGPT and Claude, can grade translations instead.

The short answer, according to new research from Apple ML Research: not reliably enough, not yet.

What did the researchers actually test?

They compared AI graders against human ones on two quality-checking systems. MQM, short for Multidimensional Quality Metrics, asks reviewers to find specific errors in a translation and label what type each one is, a mistranslation versus a fluency problem, for example. ESA, Error Span Annotation, is a simpler scheme that just asks reviewers to mark which parts of a translation are wrong.

Tests ran across several language pairs and domains, measuring how often the AI graders matched what human experts said. Agreement was modest. The gap between AI and human graders was similar in size to the gap between two human graders working independently. That finding matters: AI isn't dramatically worse than humans, but it's not a clean substitute either, because human-to-human disagreement on these tasks is already high enough to be a problem.

Fine-grained labelling, deciding not just that an error exists but precisely what kind it is, proved hardest for both groups. That's also the most useful information for a translation team trying to fix systematic problems in an automated pipeline.

Does this mean AI translation checkers are useless?

No. For broad-brush quality screening, AI graders hold up reasonably well. The trouble comes when you need detailed, reliable error labels, the kind that tell a developer whether their system is consistently mangling verb tenses or dropping named entities.

Apple ML Research has been busy on evaluation questions: our earlier report on 11 September covered the team's method for scoring AI video captions on factual accuracy rather than surface word overlap, and this translation study extends that same scepticism to a different annotation task. The study on AI behaviour and writing quality showed that optimising models on one dimension often degrades another. Fine-grained translation grading is one more place where that ceiling is lower than the industry would like.

For organisations running high-volume translation workflows, the practical read is this: AI graders can handle first-pass screening to filter out obviously poor translations. For decisions that require knowing exactly what went wrong, human reviewers are still the more dependable option.

Common questions

Why is grading a translation so hard, even for AI?

Translation quality is partly subjective, and two expert humans often disagree on whether a phrasing is a real error or just a stylistic choice. An AI model inherits that ambiguity and has no special ability to resolve it.

Should companies stop using AI to check translations?

Not necessarily. The research suggests AI works acceptably for quick, coarse-grained screening. It's precise, category-level error labelling where AI graders are still too inconsistent to replace trained human reviewers entirely.