OpenAI Built an AI That Hacks Its Own Models to Make Them Safer

Key points

• OpenAI created GPT-Red, an AI system trained to attack other AI models, and announced the project this week.
• GPT-Red found that more than 90% of its strongest attacks worked against GPT-5, released in August 2024, compared to fewer than 23% against the new GPT-5.6.
• GPT-Red discovered a previously unseen attack type that researchers call a "fake chain of thought."
• OpenAI tested GPT-Red against a real vending machine agent called Vendy and the system successfully changed prices and cancelled customer orders.
• OpenAI will not release GPT-Red to the public.

OpenAI has built an AI whose entire job is to break other AIs. The system, called GPT-Red, acts as a kind of sparring partner: it relentlessly tries to hack OpenAI's own chatbot models so the company can patch the holes before anyone else finds them.

The idea comes from an old security practice called red-teaming, where a dedicated group of people try every trick they know to break a system. GPT-Red automates that process, running attacks far faster and more persistently than a human team could manage on its own.

The timing matters. OpenAI announced GPT-Red alongside last week's release of GPT-5.6, the latest version of its flagship AI model. The company says GPT-5.6 is its hardest model to break yet, in part because it was trained against GPT-Red's attacks.

Research scientists Nikhil Kandpal and Dylan Hunn, who co-created the system, explain that the threat to AI models is growing quickly. As AI gets used in more places, especially in the form of AI agents (software that can browse websites, read emails, edit files, and interact with other programs on your behalf) the number of ways an attacker could cause harm grows with it.

GPT-Red was built by putting an untrained model in what researchers call a self-play loop with several other models. It played attacker; they played defender. Round after round, every side got better. Think of it as two chess players who only ever play each other, grinding away until both are formidable.

Most of the team's effort went into defending against a specific threat called prompt injection. This is where an attacker hides secret instructions inside text that an AI reads, such as a webpage or an email, tricking the AI into doing something its user never asked for, like leaking private information or sabotaging a document.

What's a fake chain of thought, and should that worry ordinary people?

Yes, modestly, and here is why. A chain of thought is the internal scratchpad an AI uses to work through a problem step by step. GPT-Red found a way to slip a forged note into that scratchpad, convincing the target AI that it had already checked something it never actually checked. Research scientist Chris Choquette-Choo compared it to being told that 1+1=3 and that you already confirmed it yourself. The model just accepts it and moves on.

For a regular person using an AI assistant, this kind of attack would not come from you. It would hide inside a document or website the AI reads on your behalf. The AI might then act on false information without any warning.

GPT-Red has real limits. It struggles with attacks that require back-and-forth conversation, and it is not yet reliable at using images to carry hidden instructions. Human red-teamers still catch things it misses.

Jessica Ji, a senior research analyst at Georgetown University's Center for Security and Emerging Technology, reviewed the work and called the results promising.

OpenAI will not publish GPT-Red. Researchers told MIT Technology Review the project took over a year and enormous computing resources to build, making a quick copycat unlikely.

What to watch for if you use AI tools at work: Be cautious when an AI agent reads external content (emails, documents, web pages) and then takes action on your behalf. If the output looks wrong or unexpected, treat it as a possible sign that the AI read something it should not have trusted.