Voice-to-Wheels: Turning Speech into ROS 2 Motion

A robot you control by voice lives and dies on two numbers: how long from speaking to moving, and how often it does the wrong thing. Our first version was slow and occasionally wrong, and the fix was not a better model, it was a better structure. Here is how we ended up building voice-to-wheels.

Key Insights

• The voice chain is speech-to-text, then intent, then a ROS 2 motion command, and on our robot all of it runs on-device
• Routing every command through a language model was both slower and occasionally wrong, for example mapping “move forward” to a sideways strafe
• A deterministic rule layer for the known command vocabulary runs instantly and reliably, with the model kept only as a fallback for open phrasing
• Safety commands like stop need their own fast path that bypasses transcription entirely
• The right structure matters more than the model choice for a responsive, trustworthy voice interface

The naive version, and why it disappoints

The obvious design is a pipeline: transcribe the speech, hand the text to a small on-device language model to figure out what the person meant, and send the resulting action to the robot. It works, and it disappoints in two specific ways.

It is slow, because the language step adds one to three seconds on top of transcription for even a trivial command. And it is occasionally wrong in ways that feel dumb to a user: our model would take “move forward” and produce a sideways strafe, or take “come here” and produce nothing. When a person says a plain command and the robot does something odd, the whole thing feels broken, no matter how clever the model is elsewhere.

The fix: a rule layer in front of the model

The insight is that the commands that matter most for a demo or a product are a small, known vocabulary: stop, forward, back, turn, follow, play. Those do not need a language model at all. We put a deterministic rule layer in front of the model that resolves the known vocabulary directly, in effectively zero time, and only hands the sentence to the language model when nothing in the rules matches.

That single change fixed both problems. The core commands became instant and correct, because they no longer depended on a model guessing at them, and the language model was reserved for the open-ended phrasing it is actually good at. The user-facing result is a robot that responds to plain commands immediately and predictably, which is the whole point.

Stop is special

There is one more structural rule worth stating on its own: a safety command cannot wait in line. On our robot a spoken “stop” that went through the full transcription pipeline took several seconds, which is far too long when the robot is moving toward a table edge. So stop got its own path that bypasses transcription entirely and publishes the halt directly, alongside a dedicated stop control on the dashboard. The direct path responds in well under a tenth of a second versus seconds for the spoken route. When it matters, you use the fast path.

The general principle: the more important and time-critical a command is, the fewer stages it should have to pass through.

What we keep for a conversation

This post is the shape of the design, not the implementation. The reason is that the interesting decisions for your product, where the rule layer ends and the model begins, how the intent maps to your specific actions, how you keep it responsive on your hardware, are judgement calls we make with you, not a snippet to paste. Structuring a voice interface that is fast and trustworthy on-device is the work we do.

If you are building voice or command interfaces on an edge robot or device, on the MediaTek Genio or on NVIDIA Jetson, tell us what you are building and we will help you get the structure right. For the speech model behind this, see Whisper on the Genio NPU, and for the whole robot, the ROSOrin build overview.