Talk to Your Self-Driving Car: Researchers Build a System That Lets You Ask for a Smoother Ride

Key points

• Researchers at Delft University of Technology built a system that translates plain-English passenger requests into adjustments to a self-driving car's driving behaviour.
• The system uses GPT-4o-mini, OpenAI's compact language model, to interpret requests and tune how the car's motion planner weighs speed, smoothness and safety.
• All changes are described to the passenger in plain language and require confirmation before taking effect.
• The work was tested only in simulation, using a virtual highway-merging scenario, not on public roads.
• Independent experts praise the safety architecture but warn that formal mathematical guarantees of safe behaviour are still missing.

A self-driving car that you can actually boss around. That is the idea behind new research from Delft University of Technology (TU Delft) in the Netherlands, covered first by IEEE Spectrum AI.

The team built a system that lets a passenger type or speak a request, such as "I am running late, go fast" or "I feel carsick, please drive more gently", and have the car's software adjust its driving style accordingly.

How does it actually work?

The system sits between you and the car's motion planner, the piece of software that chooses a safe, efficient path through traffic. Normally engineers tune that planner before the vehicle ever reaches a road, and passengers have no say.

The new layer adds GPT-4o-mini, a compact version of OpenAI's large language model (the technology behind chatbots like ChatGPT), to act as a translator. You speak; the model works out which driving parameters, things like steering smoothness, acceleration, and speed, should shift up or down. It then presents the planned changes in plain English and asks for your approval before doing anything.

Critically, the language model never directly steers the car. It only adjusts the weights inside the motion planner, a mathematical framework that still enforces a safe baseline the engineers set in advance. Think of it as the language model turning knobs within a locked cabinet, not driving the car.

"The motion-planning problem is not only about reaching a place while avoiding collisions, it's also how you do it," said lead author Diego Martinez-Baselga, a postdoctoral researcher at TU Delft. "The motivation here is trying to make the way the autonomous car drives adaptable by end users easily, just by talking to the car."

What did the tests actually show?

The researchers ran simulations inside nuPlan, a widely used virtual environment for testing self-driving software, focusing on the tricky scenario of merging onto a busy highway. Across eight different passenger prompts, the system adjusted parameters in the direction the passenger intended. Requests for comfort produced smoother, gentler driving. Requests for urgency produced higher speeds.

That is promising. It is also, for now, only a simulation with eight prompts.

Should passengers trust it?

Not yet on real roads, and two outside experts spell out why.

Nicolas Baumann, a PhD student at ETH Zurich who published similar research last year using a language model to tune a model racing-car controller, likes the safety architecture. Because the language model is kept separate from the core controller, even if the model "hallucinates" (produces a confident but wrong answer), it cannot do anything physically dangerous. "You get the possibility of language interaction, but you can guarantee that it is going to be within the constraints of this classical controller," he said.

But Matthias Althoff, a professor of cyberphysical systems at the Technical University of Munich, points to a gap. His own group checks every language-model suggestion against a formal mathematical proof before the car commits to it. The TU Delft system has no such check. "As with any LLM, it is not guaranteed that the result is correct," Althoff said. "For that reason, we safeguard the decisions of the LLM in our works."

The TU Delft team is presenting the work at the IEEE Intelligent Transportation Systems Conference in September. A preprint is available on arXiv.

What happens next?

For passengers, nothing changes soon. The research is at a very early stage, and the gap between a simulation on a virtual highway and a certified, road-legal vehicle is measured in years of engineering and regulation.

Still, the concept matters. Most people are uneasy about handing full control to a computer they cannot talk to. A system that listens, explains itself, and asks permission before changing behaviour is a small but real step toward autonomous vehicles that feel less like a locked box and more like a driver you can actually work with.

Common questions

Could someone use this to make a self-driving car go dangerously fast?

No, not by design. The language model only adjusts values within a safe range that engineers fix in advance. A passenger asking for reckless speed would hit the ceiling of what the system permits.

Is this available in any real cars today?

No. The system exists only in simulation. It has not been tested on public roads, and no car manufacturer has announced plans to adopt it.

What is a motion planner and why does it matter?

A motion planner is the software brain that decides every moment how a self-driving car should move: how fast, which lane, when to brake. Letting passengers nudge it through conversation is the core idea of this research.