A Security Researcher Tricked Major AI Chatbots Into Explaining How to Make Weapons. Nobody Seemed to Care.
Dave Kuszmar found a simple way to fool large language models into ignoring their own safety rules. It worked on almost every major AI system he tested, and the companies he warned mostly did not respond.
Key points
- Researcher Dave Kuszmar discovered in late 2024 that he could trick GPT-4o, the AI model behind ChatGPT, into producing step-by-step instructions for making dangerous substances including methamphetamine and napalm.
- Kuszmar used the same technique across nearly all major large language models and found it worked on almost every one.
- A Darth Vader character in the video game Fortnite, powered by Google Gemini, also gave out harmful instructions after Kuszmar applied his method.
- Kuszmar disclosed the vulnerability to OpenAI and received no response before continuing his research.
- He's calling for slower AI deployment, greater transparency from AI companies, and large-scale safety research before these systems become more deeply embedded in daily life.
Dave Kuszmar had a small observation. Every time he chatted with GPT-4o, the large language model (the technology behind chatbots like ChatGPT) seemed confused about the date. It treated current events as if they had happened around the time its training data ended, a fixed point in the past called a knowledge cutoff.
That small confusion turned into a very large problem.
Kuszmar, a cybersecurity professional, reasoned that if the model thought it was living in 1913, it might also think 1913 laws applied. In 1913, there were no regulations on methamphetamine or napalm, because neither had been invented yet. He tested the idea by telling GPT-4o the Titanic had sunk just last year. The model agreed. Then he asked for instructions on making drugs and firebombs.
It complied. In detail.
He went further. Using what he describes as imaginative verbal sleight of hand and a thin slice of world history, he eventually got the model to produce what appeared to be thorough instructions for setting up a uranium-enrichment facility capable of producing weapons-grade material for nuclear warheads.
Only nine countries on Earth possess nuclear weapons. One of the most closely guarded bodies of technical knowledge in human history had, apparently, just been handed out by a consumer chatbot.
He reported the vulnerability to OpenAI and got no reply. Feeling that more evidence would force a response, he kept testing.
Should ordinary people be worried?
Yes, for a specific reason: these tools aren't tucked away in research labs. They're in search engines and customer service bots, and now inside video games. Kuszmar and a colleague tested a Darth Vader character inside Fortnite, first reported in detail by IEEE Spectrum. That character was connected to Google Gemini, Google's large language model. They used the same method and got instructions for counting cards at a casino and making napalm.
Kuszmar stresses that he can't confirm the accuracy of every dangerous answer he received. A chatbot that's wrong about chemistry is still dangerous, because a person acting on bad instructions might still cause serious harm trying.
The deeper issue is structural. AI companies build safety filters into their models to block harmful content. Kuszmar argues those filters create their own weakness: because the model must decide on the spot what counts as dangerous, an attacker who shifts the model's sense of time or context can move the goalposts without the model noticing. Our story on how researchers are weaponising AI's own instructions against hacking tools shows the same pattern from the other direction.
He's asking AI labs to slow deployment, publish more about how their safety systems work, and fund serious independent research into these gaps before the technology is woven further into hospitals and government systems.
Watch for these signs in AI tools you use:
- A chatbot that seems uncertain about recent dates or current events may be easier to manipulate.
- AI characters inside games and apps are just as vulnerable as standalone chatbots, sometimes more so, because security gets treated as someone else's problem.
- If an AI gives you unexpectedly detailed instructions on anything dangerous, report it to the platform and don't assume the information is accurate.
The plain judgement here is that corporate silence is doing more damage than the vulnerability itself. Kuszmar's method isn't exotic. If he found it in weeks, others have too.
