Talk to Your Self-Driving Car: Researchers Build a System That Lets You Ask for a Smoother Ride

Key points

• Researchers at Delft University of Technology built a system that translates plain-English passenger requests into adjustments to a self-driving car's driving behaviour.
• The system uses GPT-4o-mini, OpenAI's compact language model, to interpret requests and tune how the car's motion planner weighs speed, smoothness and safety.
• All changes are described to the passenger in plain language and require confirmation before taking effect.
• The work was tested only in simulation, using a virtual highway-merging scenario, not on public roads.
• Independent experts praise the safety architecture but warn that formal mathematical guarantees of safe behaviour are still missing.

A self-driving car you can actually boss around. That is the idea behind new research from Delft University of Technology (TU Delft) in the Netherlands, covered first by IEEE Spectrum AI.

Passengers type or speak a request, such as "I am running late, go fast" or "I feel carsick, please drive more gently", and the car's software adjusts its driving style accordingly. As we've been tracking the commercial push toward autonomous vehicles, the question of passenger trust keeps surfacing; this research takes a direct run at it.

How does it actually work?

The system sits between you and the car's motion planner, the software that chooses a safe, efficient path through traffic. Normally engineers tune that planner before the vehicle ever reaches a road, and passengers have no say.

GPT-4o-mini, a compact version of OpenAI's large language model (the technology behind chatbots like ChatGPT), acts as a translator. You speak; it works out which driving parameters should shift. It then presents the planned changes in plain English and asks for your approval before doing anything.

The language model never directly steers the car. It only adjusts the weights inside a model predictive-path integral controller, a framework the TU Delft team had previously developed, which rates candidate paths on criteria including speed, steering angle and collision probability, then picks an optimal blend. The language model turns knobs within that system; it does not drive.

"The motion-planning problem is not only about reaching a place while avoiding collisions, it's also how you do it," said lead author Diego Martinez-Baselga, a postdoctoral researcher at TU Delft. "The motivation here is trying to make the way the autonomous car drives adaptable by end users easily, just by talking to the car."

The scenario description the model currently receives was hand-written by the researchers, though Martinez-Baselga says it could ultimately come straight from the vehicle's own perception system.

What did the tests actually show?

The team ran simulations inside nuPlan, a widely used virtual environment for self-driving software, focusing on highway merging. Across eight passenger prompts, the system adjusted parameters in the direction intended: smoother driving for comfort requests, higher speeds for urgency.

Promising. Also, for now, eight prompts in a simulator.

Should passengers trust it?

Not yet on real roads, and two outside experts explain why.

Nicolas Baumann, a PhD student at ETH Zurich who published similar work last year using a language model to tune a model racing-car controller, likes the safety architecture. Keeping the language model separate from the core controller means that even if the model hallucinates (produces a confident but wrong answer), it cannot do anything physically dangerous. "You get the possibility of language interaction, but you can guarantee that it is going to be within the constraints of this classical controller," he said. He adds, though, that setting those constraints requires considerable engineering work.

Matthias Althoff, a professor of cyberphysical systems at the Technical University of Munich, points to a harder gap. His group checks every language-model suggestion against a formal mathematical proof before the car commits to it. TU Delft's system has no such check. "As with any LLM, it is not guaranteed that the result is correct," Althoff said. "For that reason, we safeguard the decisions of the LLM in our works."

That distinction matters a lot. Bounding the model's output to a safe range is not the same as proving the output is safe, and Althoff's point is that the field will eventually need the latter.

The team presents the work at the IEEE Intelligent Transportation Systems Conference in September. A preprint is available on arXiv.

What happens next?

For passengers, nothing changes soon. The gap between a simulator and a certified, road-legal vehicle is measured in years of engineering and regulation, a reality visible in Zoox's slow march toward federal approval even with a locked design and a factory behind it.

Still, the concept addresses something real. Most people are uneasy handing full control to a computer they cannot talk to. A system that listens, explains itself and asks permission before changing behaviour is a genuine step toward vehicles that feel less like a locked box.

Common questions

Could someone use this to make a self-driving car go dangerously fast?

No, not by design. The language model only adjusts values within a safe range that engineers fix in advance. A passenger asking for reckless speed would hit the ceiling of what the system permits.

Is this available in any real cars today?

No. The system exists only in simulation. It has not been tested on public roads, and no car manufacturer has announced plans to adopt it.

What is a motion planner and why does it matter?

A motion planner is the software that decides every moment how a self-driving car should move: how fast, which lane, when to brake. Letting passengers nudge it through conversation is the core idea of this research.