OpenAI Built an AI That Hacks Its Own Models to Make Them Safer
GPT-Red is an automated red-teaming system that attacks OpenAI's own chatbots to find weak spots before real attackers do. It already discovered a trick that human testers had missed.
Key points
- OpenAI created GPT-Red, an AI system trained to attack other AI models, and announced the project this week.
- When tested, more than 90% of GPT-Red's strongest attacks worked against GPT-5, released in August 2024, compared to fewer than 23% against the new GPT-5.6.
- GPT-Red discovered a previously unseen attack type that researchers call a "fake chain of thought."
- OpenAI tested GPT-Red against a real vending machine agent called Vendy; the system successfully changed prices and cancelled customer orders.
- OpenAI will not release GPT-Red to the public.
OpenAI has built an AI whose entire job is to break other AIs. GPT-Red acts as a sparring partner, relentlessly trying to hack OpenAI's own models so the company can patch the holes before anyone else finds them.
Red-teaming is an old security practice where a dedicated group tries every trick it knows to break a system. GPT-Red automates that process, running attacks far faster than a human team could manage alone.
The timing matters. OpenAI announced GPT-Red alongside last week's release of GPT-5.6, its latest flagship model, which we've covered eight times since 13 July, most recently in the context of ChatGPT Work. The company says GPT-5.6 is its hardest model to break yet, partly because it was trained against GPT-Red's attacks.
Research scientists Nikhil Kandpal and Dylan Hunn co-created the system. As AI gets deployed more widely, especially as agents (software that can browse websites, read emails or interact with other programs on your behalf) the attack surface expands sharply. "The risk surface grows and the blast radius also grows," Kandpal told MIT Technology Review.
To build GPT-Red, the team placed an untrained model in a self-play loop with several other models: it played attacker, they played defender. Round after round, both sides improved. Think of two chess players who only ever face each other, grinding until both are formidable.
Most of the effort focused on defending against prompt injection, where an attacker hides secret instructions inside text an AI reads, such as a webpage or a document, tricking it into leaking private data or corrupting a file.
What's a fake chain of thought, and should that worry ordinary people?
Yes, at least a little. A chain of thought is the internal scratchpad an AI uses to work through a problem step by step. GPT-Red found a way to slip a forged entry into that scratchpad, convincing the target AI it had already verified something it never checked. Research scientist Chris Choquette-Choo put it this way to MIT Technology Review: "It's like if I told you that 1+1=3 and that you have verified this already. The model's like, 'Oh, okay, of course.'" It just moves on.
For a regular person, this attack wouldn't come from you. It would hide inside a document or website the AI reads on your behalf, and the AI might then act on false information without flagging anything. Our 15 July report on MemGhost showed the same basic danger from a different angle: a single poisoned email can plant a false fact inside an AI agent's long-term memory.
GPT-Red has real limits. It struggles with attacks that require back-and-forth conversation, and it isn't yet reliable at using images to carry hidden instructions. Human red-teamers still catch things it misses.
Jessica Ji, a senior research analyst at Georgetown University's Center for Security and Emerging Technology, called the results promising. OpenAI won't publish GPT-Red. Choquette-Choo noted it took more than a year and substantial computing resources to build, making a quick copycat unlikely.
Should you worry about AI agents reading your documents?
Be cautious when an AI agent reads external content and then acts on your behalf. If its output looks wrong or unexpected, treat that as a possible sign it trusted something it shouldn't have. The fake chain-of-thought attack is a reminder that an AI can be deceived before you ever see the result.
The deeper story here isn't the hacking tool itself. It's that OpenAI needed to build one at all, which says a lot about how fast the attack surface around AI agents is growing.
