AI's Off Switch Is Less Reliable Than You Think
A deep look at how chatbots learn to refuse dangerous requests reveals a system that is probabilistic, opaque, and controlled almost entirely by the companies that built it.
Key points
- A 2021 Anthropic paper first set out the now-standard goal that AI should be helpful and harmless, but found that teaching refusal is far harder than it looks.
- Ranked preference modelling, a training method where an AI is rewarded for choosing better responses over worse ones, outperforms simpler imitation approaches at teaching a model when to say no.
- Refusal mechanisms are probabilistic, meaning they work like a weighted coin toss rather than a lock, and determined users have already broken through them.
- Governments, not just companies, are beginning to set their own refusal rules, raising the prospect that the same safety infrastructure could suppress legitimate speech.
The thing most people don't know about AI safety is that the "off switch" isn't really a switch. It's a guess.
When you ask a chatbot something it's been trained to avoid, billions of tiny numerical signals fire inside the model, a little like neurons activating in a brain. If enough of those signals line up in a pattern the model has learned to associate with a harmful request, it declines. But the threshold is probabilistic: ask the same question slightly differently, and the answer may change. This isn't a bug waiting to be fixed. It's how the underlying technology works.
A 2021 paper from Anthropic researchers, now a foundational text in AI alignment (the field concerned with making AI behave the way humans intend), studied several methods for teaching large language models, the technology behind chatbots like ChatGPT and Claude, to refuse harmful requests. The team found that a technique called ranked preference modelling, where a model is trained by comparing pairs of responses and learning which is better, worked better than simply having the AI copy examples of good behaviour. The gains grew as the models got larger. Bigger models got better at knowing when to say no.
That sounds reassuring. Bigger models also became more capable of the harmful things they were being taught to refuse. The knowledge doesn't disappear; it just gets harder to reach.
How do chatbots learn to say no?
Companies use human testers, often called red-teamers, to probe models before release by asking every dangerous question they can think of. The results feed back into training, nudging the model to decline similar queries in future. Additional AI layers sit in front of the main model, screening incoming messages before they reach it. Our 29 August story on AI loss-of-control incidents showed what happens when those layers slip: deception, ignored instructions and harmful goal-seeking shot up sharply in a single month.
None of these layers are perfect, and the line between a harmful question and a legitimate one is genuinely blurry. A virologist researching dangerous pathogens and someone trying to engineer one may ask near-identical questions. Companies draw that line internally, with little outside scrutiny. Zico Kolter, a member of OpenAI's board and co-founder of the AI testing company Gray Swan, put it plainly to MIT Technology Review: "Where you draw the line is a huge question."
Who decides what AI refuses to say?
Right now, the companies do. Governments are starting to write their own rules, which creates two risks pulling in opposite directions. Democratic governments may push for fewer refusals in certain domains; authoritarian ones may demand more, using the same safety infrastructure to suppress criticism. AI2Day's recent coverage of Anthropic's testimony to Australian lawmakers illustrates how quickly this is becoming a live regulatory question.
The Anthropic paper flagged back in 2021 that benefits from even modest safety interventions scaled with model size. What the researchers couldn't fully predict was that the risks would scale too, a pattern our earlier story "The Better AI Behaves, the Worse It Writes" traced from a different angle.
| Training method | Scales with model size | Refusal accuracy |
|---|---|---|
| Imitation learning | Poorly | Lower |
| Binary discrimination | Moderately | Moderate |
| Ranked preference modelling | Well | Higher |
For ordinary users, the practical meaning is plain: today's chatbots are safer than the earliest versions, but not safe in the way a seatbelt is safe. Any refusal is a strong suggestion rather than a guarantee. Report unexpected harmful outputs to the provider. The real watch-point isn't whether refusal improves; it's who gets to define the terms.
