Researchers Are Turning Hackers' Favourite AI Weapon Against Them

Key points

• Tracebit researchers published findings on Monday showing that prompt injections can be used as a defensive trap against AI hacking agents.
• The technique works by placing hidden instructions next to passwords and cryptographic keys stored on Amazon Web Services (AWS), Amazon's cloud computing platform.
• Hidden instructions trick an attacking AI into attempting something its own safety rules forbid, causing it to stop.
• No patch or software update is needed on the attacker's side; the AI's built-in safety barriers do the heavy lifting.

For years, attackers have used a trick called prompt injection to hijack AI assistants. A prompt injection is a hidden instruction buried inside ordinary content, like an email or a calendar invite, that tells an AI to do something its owner never intended. The AI reads it, treats it as legitimate, and follows it. Sensitive data leaks out.

Now a security firm called Tracebit has flipped the trick around.

On Monday, Tracebit researchers reported, first covered by Ars Technica AI, that they'd found a way to use prompt injections as a trap rather than a weapon. Place a carefully worded hidden instruction right next to the passwords, cryptographic keys (long strings of characters that encrypt and protect data), and other secrets stored on AWS. When an AI-powered hacking agent, software that carries out multi-step attacks on its own, breaks in and reads those credentials, it also reads your instruction.

That instruction tells the attacking AI to do something its developers have specifically forbidden.

AI companies build what are called guardrails into their models: a set of rules that stops the AI from crossing certain lines, like helping someone cause harm. When the trapped AI tries to follow the injected command, it hits one of those lines and stops. The attack ends.

It's an elegant piece of security judo.

The approach belongs to a broader category called honeytokens or canary tokens, decoy credentials left in places attackers are likely to look. If anyone touches them, you know a breach is under way. Tracebit's version does the same job and actively disrupts the attacker, rather than just sounding an alarm.

Does this mean AI hacking is solved?

No. This technique works specifically against AI agents that rely on large language models, the technology behind tools like ChatGPT, because those models have built-in safety rules that can be triggered. A human attacker, or a stripped-down AI with no guardrails, would ignore the injected instruction entirely. The finding is promising, but it's one layer of defence, not a complete answer.

Watch the guardrails question closely. If AI developers trim safety barriers under commercial pressure, the entire premise of this defence weakens. What Tracebit has shown is genuinely clever; whether it stays useful depends on decisions being made well outside the security community.