AI guardrails built to stop hackers are now blocking the people trying to stop hackers

Key points

• Anthropic's AI models Mythos and Fable faced U.S. government export controls in June, restricting who could access them outside the country.
• Both Anthropic and OpenAI run vetted programmes that give approved cybersecurity researchers slightly looser access to their AI tools.
• Multiple offensive security researchers say the filters still block essential work and produce unpredictable results even inside those programmes.
• Some researchers say the restrictions push them toward Chinese open-source AI models, such as GLM, that carry no filters or vetting at all.
• One researcher at a smartphone-component maker said his team's AI tools became nearly useless for security work because no one had applied for the vetted programme.

Cybersecurity has always been a two-sided game. The same knowledge that lets a defender patch a hole lets an attacker crawl through it. Now AI tools have landed in the middle of that tension, and the safety filters built into them are starting to frustrate the people they were meant to protect.

What is actually going wrong?

AI companies put guardrails, meaning automatic rules that stop the software from answering questions they consider dangerous, on their models to prevent criminals from using them to build malware or plan attacks. The problem is that those same rules block legitimate security researchers from doing their jobs.

Chris Anley, chief scientist at security consulting firm NCC Group, gave a clean example. When he asks an AI model to try to exploit a software bug, he is testing whether the bug is real and dangerous enough to fix. A guardrail that refuses that request does not protect anyone. It just leaves the flaw unconfirmed and potentially open to a real attacker.

"Fix this code is both an essential mechanism for defence but also a roadmap for finding critical vulnerabilities," Anley said. "The same tool is both an offensive tool and a defensive tool, and the two can't really be unpicked."

His analogy: it is like a hammer. You cannot build a house without one, but it can also break a window.

How bad is the inconsistency?

It is patchy enough to waste serious time. Chris Thompson, chief executive of cybersecurity firm RemoteThreat and founder of Offensive AI Con, a conference focused on security and AI, told TechCrunch that even inside Anthropic's and OpenAI's approved researcher programmes the guardrails fire differently from one day to the next.

"You spend a lot of time negotiating with the model instead of working on the core security programme," Thompson said. "Instead of analysing a vulnerability, you're trying to find why models are over-sanitising the output."

One unnamed researcher at a smartphone-component manufacturer put it plainly: "If it catches wind we're doing anything security related, it just stops and isn't usable."

Who is this pushing people toward?

Open-source models with no filters at all, many of them developed in China. Thompson said responsible researchers are being steered away from systems subject to U.S. oversight and toward foreign-built tools that carry no accountability.

"You have these responsible researchers that are being pushed away from U.S.-governed systems to foreign-owned systems," he said. "I think it's more harmful than good to have these guardrails in place."

Not every researcher feels blocked. Giuseppe Cali, who finds and develops zero-days (previously unknown software flaws that no patch yet exists for), said he keeps AI away from the actual vulnerability-hunting because he wants that discovery to stay his own. For supporting tasks like understanding unfamiliar code, he finds the tools useful enough.

"I am jealous of my bugs," Cali said, "and I like this game too much to let models play it for me."

Thompson's fix is not to tear the guardrails out entirely. He wants AI labs to widen access for vetted researchers, apply consistent rules, and hold bad actors accountable when abuse happens. Without that balance, he argues, defenders will fall behind the attackers who face no such restrictions.

Common questions

Do these restrictions affect ordinary computer users?

Not directly. The friction described here hits professional security researchers, not everyday people using AI for writing or work tasks. General guardrails remain in place for everyone.

Why do AI companies restrict cybersecurity questions at all?

They worry that detailed, step-by-step attack guidance could help criminals build malware or breach systems. The difficulty is that the same guidance is exactly what legitimate defenders need to test and harden those systems first.

What is an open-source AI model and why does it have no guardrails?

An open-source AI model is one whose underlying code is published freely for anyone to download and run on their own computer. Because it runs locally, there is no company in the middle to enforce rules, which is why researchers use them for sensitive work but also why they carry risks of their own.