AI Fixed Security Bugs Correctly Just 26% of the Time in a 6,080-Patch Study
1Password tested two leading AI models on real software flaws and found that most patches looked fine but weren't. What that means for the software you use every day.
Key points
- 1Password tested 6,080 AI-generated security patches and found only 26% fixed the flaw without unintended side effects.
- Both ChatGPT-5.5 and Claude Opus 4.8 produced hidden-defect patches in 53.9% of cases when the vulnerability was complex.
- Nearly half of all patches, 49.3%, left at least one path open that an attacker could still exploit.
- Each patch attempt cost between $2.11 and $2.81, but verifying whether the fix is safe requires expert human time on top of that.
- Anthropic, the company behind Claude, told 1Password that human specialists must remain the final sign-off at current AI capability levels.
1Password, the password-management company most people know from their phones, published a study this week with a sobering finding: AI tools are not reliably fixing the security bugs they claim to fix. As first reported by ThreatVectr, the research ran 6,080 patches generated by two widely used AI models against six real, recently disclosed software vulnerabilities.
Only 26% of patches solved the problem cleanly, with no hidden damage.
What exactly went wrong with the patches that failed?
Most bad patches looked fine on the surface. They compiled, meaning the computer accepted the code, passed basic tests, and showed no obvious error. The problem was buried underneath.
Researchers named this category of bad output FLAWED, short for Fix-Like Artifacts With Embedded Defects. The code appears healthy but carries a hidden weakness.
Researcher Keith Hoodlet gave a clear example. When both models tackled a flaw in Spring AI, a software framework developers use to build AI-powered applications, they filtered out the specific attack characters used in the test. An attacker using slightly different characters would sail straight through the same old hole. More than a third of initially promising patches were fragile in exactly this way.
The numbers break down like this:
| Outcome | Share of patches |
|---|---|
| Fixed cleanly, no side effects | 26.0% |
| Left at least one attack path open | 49.3% |
| Hidden defects in complex flaws | 53.9% |
| Fixed original flaw, added a new one | 2.3% |
| Failed to fix original and added a new flaw | 2.2% |
Should you worry about the apps you use today?
Not immediately, but software companies are under constant pressure to ship security fixes fast, and AI tools are increasingly part of that process. We've been tracking this pattern closely: our 6 August story on Meta's AI hacking a company during a security test showed what can go wrong when AI acts without sufficient human oversight.
If AI patches skip careful human review, the fix that lands on your phone or laptop may not actually protect you. The code looks patched. It might not be.
Anthropics position is clear: human domain experts must remain the final reviewers. Most professional security analysts already treat AI output as a first draft, not a finished product. That habit matters more now.
Cost is part of the picture too. Each patch cycle ran about $2.11 with ChatGPT-5.5 and $2.81 with Claude Opus 4.8. Cheap per attempt. But Hoodlet's point is that the real cost is the skilled human time needed to check whether any given patch is actually safe to ship. That time doesn't disappear just because a machine wrote the code.
The honest read on this study: speed is being sold as a feature, but 26% accuracy isn't a feature, it's a liability. If you manage software or work in IT, treat every AI-generated security patch the way you'd treat a junior developer's first pull request. Read it. Test it properly.
