A New Security Benchmark Pits AI Against Real Code Vulnerabilities. Two Models Are Nearly Tied at the Top.
Collinear AI's CWE-bench v2 tests whether AI can find and fix security flaws in actual software. Gemini 4 Argon leads, but Claude Opus 5.5 is right behind it, and even the best model still fails roughly one in five tasks every single time.
Key points
- Gemini 4 Argon scored 67.8% on CWE-bench v2, the highest of 11 models tested, with Claude Opus 5.5 scoring 67.3% and finishing second.
- The benchmark covers 91 different categories of software security weakness across 100 tasks drawn from real open-source code repositories.
- DeepSeek V4.1 Flash completed tasks for just $0.04 per attempt, compared to $1.83 for Claude Opus 5.5 and $3.92 for Gemini 4 Argon.
- Even the top-ranked model failed 19% of tasks on every single attempt, meaning a real share of vulnerabilities still evade AI detection.
- Collinear AI published the benchmark and its scoring tools publicly on Hugging Face, so any team can test their own model.
Hand a software engineer a messy real-world codebase and say: find the security hole and fix it without breaking anything else. That's roughly what CWE-bench v2 asks of AI models, and the results published this week by Collinear AI show how close the frontier has come, and how far it still has to go.
CWE stands for Common Weakness Enumeration, a standardised catalogue of software security flaws. A CWE might describe a flaw where a programme trusts input it shouldn't, or where memory is handled carelessly in a way an attacker could exploit. An AI agent, meaning a piece of software that can carry out multi-step tasks on its own, receives a snapshot of a real open-source project and a plain-language instruction. It must find the flaw, patch it, and leave everything else intact.
Who came out on top?
Gemini 4 Argon leads the table at 67.8%, fixing the right vulnerability on its first attempt roughly two thirds of the time. Claude Opus 5.5, Anthropic's latest flagship, sits just behind at 67.3%. AI2Day has been tracking Anthropic's rapid commercial growth this month, and its top model is now genuinely competitive with Google's best on a demanding, real-world task.
| Model | Pass@1 (first attempt) | Pass@4 (any of four attempts) | Cost per attempt |
|---|---|---|---|
| Gemini 4 Argon | 67.8% | 81.0% | $3.92 |
| Claude Opus 5.5 | 67.3% | 78.0% | $1.83 |
| DeepSeek V4.1 Flash | 57.0% | Not listed | $0.04 |
DeepSeek V4.1 Flash is the outlier. Its 57% score trails the leaders by ten points, but at four cents a task it costs a fraction of either rival. For a security team running thousands of automated checks on a tight budget, that arithmetic matters.
Still, the ceiling is the thing worth staring at. Ninety-five of the 100 tasks were solved by at least one model, confirming the problems are genuinely solvable. Yet the best model failed 19 of them on every single attempt. A defender has to get it right every time. An attacker only needs one gap.
What does this mean for real security work?
For now, these tools work best as a first pass, not a final answer. A security engineer who pipes code through an AI checker today will catch more flaws faster, but that 19% blind spot means human review isn't optional. That pattern echoes what we found covering benchmark reliability on 27 October: strong aggregate scores can mask narrow, stubborn failure modes that matter enormously in practice.
Scoring is fully programmatic, so results are reproducible without extra cost. Collinear AI also maintains a separate unpublished set of tasks that no model has trained on, which matters because models can sometimes appear to remember a correct answer rather than reason their way to one. The public tools and dataset are on Hugging Face.
