Testing AI Fairness Rules Is Easy. Proving They Hold Is a Different Problem Entirely
New research finds that a mathematical shortcut linking two kinds of AI auditing breaks down the moment you try to verify, not just test, that a system behaves consistently.
Key points
- Researchers at Apple ML Research published findings showing that testing and verification of location-invariant properties, a class of mathematical rules about how AI systems handle data, do not behave the same way.
- A relationship that made testing these properties straightforward turns out to offer no help when the goal shifts to verification, which sets a stricter standard of proof.
- The gap matters for anyone trying to certify that an AI model treats similar inputs consistently, a concern at the heart of fairness and reliability auditing.
- The work is theoretical but has practical implications for regulators and developers who rely on statistical shortcuts to audit AI behaviour at scale.
There is a quiet assumption baked into a lot of AI auditing: if two mathematical problems look structurally similar, a clever solution to one should carry over to the other. New theoretical work from Apple ML Research suggests that assumption can fail in ways that matter.
The research centres on something called location-invariant properties. That is a formal name for a simple idea: a rule about a function, a mathematical object that maps inputs to outputs, that only cares about how often each output value appears, not where it appears. Think of it like judging a playlist by how many times each song plays, regardless of the order.
It turns out that testing whether a function has this kind of property is closely related to testing whether a probability distribution, a description of how likely different outcomes are, has the matching property. Researchers had known this for years and found it useful. It meant techniques from one field could travel to the other.
But the new paper draws a hard line at verification.
Verification is stricter than testing. Testing checks a property by sampling: you look at a limited number of inputs and decide whether the rule probably holds. Verification means confirming it definitively holds, given a trusted description of the system you are checking. The researchers show that the tidy relationship between functions and distributions, so helpful in testing, falls apart completely in the verification setting.
Why does this matter outside of mathematics?
Should auditors be worried?
Yes, in a specific and practical way. Developers and regulators increasingly want to certify that AI systems behave consistently, not merely suggest they probably do. If a shortcut that works for statistical spot-checks does not work for formal guarantees, then tools and proof methods built on that shortcut may give false confidence.
The split holds in multiple technical settings the paper examines, not just one edge case. That makes it a structural finding, not a curiosity.
For now, this is theoretical research. It does not point at a broken product or a flawed audit that happened in the real world. What it does is narrow the toolkit: methods that are valid for probabilistic testing need separate, harder justification before anyone uses them to certify consistency guarantees.
For patients, customers, or workers whose lives are shaped by AI decisions, the takeaway is indirect but real. Formal verification of AI fairness rules is harder than it looks, and this paper explains precisely why.
