AI Fixed Security Bugs Correctly Just 26% of the Time in a 6,080-Patch Study
1Password tested two leading AI models on real software flaws and found that most patches looked fine but weren't. Here's what that means for the software you use every day.
Key points
- 1Password tested 6,080 AI-generated security patches and found only 26% fixed the flaw without unintended side effects.
- Both ChatGPT-5.5 and Claude Opus 4.8 produced hidden-defect patches in 53.9% of cases when the vulnerability was complex.
- Nearly half of all patches, 49.3%, left at least one path open that an attacker could still exploit.
- Each patch attempt cost between $2.11 and $2.81, but verifying whether the fix is safe requires expert human time on top of that.
- Anthropic, the company behind Claude, told 1Password that human specialists must remain the final sign-off at current AI capability levels.
1Password, the password-management company most people know from their phones, published a study this week with a sobering finding: AI tools are not reliably fixing the security bugs they claim to fix. As first reported by ThreatVectr, the research ran 6,080 patches generated by two widely used AI models against six real, recently disclosed software vulnerabilities.
The headline number is 26%. That is the share of patches that actually solved the problem cleanly, with no hidden damage.
What exactly went wrong with the other 74%?
Most of those patches looked fine on the surface. They compiled (meaning the computer accepted the code), passed basic tests, and showed no obvious error. The problem was buried underneath.
Researchers named this category of bad output FLAWED, short for Fix-Like Artifacts With Embedded Defects. The label fits: the code appears healthy but carries a hidden weakness.
Researcher Keith Hoodlet gave a clear example. When both AI models tackled a flaw in Spring AI, a software framework developers use to build AI-powered applications, they filtered out the specific attack characters used in the test. An attacker using slightly different characters would sail straight through the same old hole. More than a third of initially promising patches were fragile in exactly this way.
The numbers for the rest of the failures break down like this:
| Outcome | Share of patches |
|---|---|
| Fixed cleanly, no side effects | 26.0% |
| Left at least one attack path open | 49.3% |
| Hidden defects in complex flaws | 53.9% |
| Fixed original flaw, added a new one | 2.3% |
| Failed to fix original and added a new flaw | 2.2% |
Should you worry about the apps you use today?
Not immediately, but you should know that software companies are under constant pressure to ship security fixes fast, and AI tools are increasingly part of that process.
If those AI patches skip careful human review, the fix that lands on your phone or laptop may not actually protect you. The code looks patched. It may not be.
Anthropics position is clear: human domain experts must remain the final reviewers. Most professional security analysts already treat AI output as a first draft, not a finished product. That habit matters more now.
Cost is part of the picture too. Each patch cycle ran about $2.11 with ChatGPT-5.5 and $2.81 with Claude Opus 4.8. Cheap per attempt. But Hoodlet's point is that the real cost is the skilled human time needed to check whether any given patch is actually safe to ship. That time does not disappear just because a machine wrote the code.
One honest takeaway: if you manage software or work in IT, treat every AI-generated security patch the way you would treat a junior developer's first pull request. Read it. Test it properly. The machine is fast, but fast is not the same as right.
