Should Your AI Help You Do Anything You Ask, No Matter What?
A prominent tech founder says a properly aligned AI should help users do whatever they want, including things that are clearly illegal. The argument is wilder than it sounds, and more interesting.
Key points
- Comma AI founder George Hotz argued publicly that AI should be aligned with its individual user's wishes, including helping with illegal activities.
- Hotz compared user-aligned AI to a gun, which doesn't judge how it's used.
- The debate follows a policy paper called AI 2040: Plan A, from the AI Futures Institute, which proposed slowing AI development for 14 years.
- Most mainstream AI assistants, including ChatGPT and Claude, are centrally managed services that refuse requests deemed harmful.
- The argument raises a real question about who AI should ultimately serve: the individual user or society at large.
George Hotz, founder of Comma AI (a company that makes driver-assistance software) and a longtime figure in the hacking community, posted an argument over the weekend that's a genuine conversation starter.
His position: a properly aligned AI, one that serves its user, should help that user do whatever they ask. Order suspicious chemicals. Explain illegal processes. Help plan a crime. If the user wants it, the AI should assist.
Hotz compared this kind of AI to a gun. A gun doesn't ask questions.
He was responding to a policy paper called AI 2040: Plan A, published by the AI Futures Institute. The paper proposes that the world's AI researchers voluntarily slow development for 14 years to give humanity time to figure out safety. Hotz disagrees. He believes the real solution is local AI models, software that runs on your own device and answers only to you, rather than centrally managed services like ChatGPT or Claude that a company controls and restricts.
That part of his argument is genuinely interesting. As TechCrunch AI noted in its coverage, most AI today lives on company servers, is shaped by company rules, and can be changed or shut down at any time. A personal AI that lives on your phone and works only for you is a different thing. More like a private tool than a subscription service. AI2Day has tracked seven stories on AI alignment in the last 90 days, and the tension between user control and central oversight runs through all of them.
But Hotz pushes past that reasonable point into thornier territory. His framing is essentially: freedom means no limits.
The practical problem is that freedom hasn't worked that way for anyone. Roads, food safety rules, contract law: these are constraints that make the broader network of society function. Products used by millions of people sit inside that network whether their makers like it or not.
Here's the plain version: if an AI helps one person plan something that harms another person, the victim didn't consent to that arrangement. The first user's freedom ends somewhere near the second person's wellbeing.
Should you worry about this?
For ordinary users, none of this changes what AI tools can do today. ChatGPT and Claude will keep refusing requests that cross legal or ethical lines. Local AI models do exist and are getting better, but capable ones still require technical setup that most people would find frustrating.
The question of who controls your AI and what rules it follows will matter more and more as smaller models improve. Hotz is overstating the freedom case, but he's pointing at a real shift coming: the gap between AI as corporate service and AI as personal tool is closing, and nobody's quite figured out the rules for what happens when it does.
