Whisper on MediaTek Genio NPU: Our Deployment Story

We got a call about putting OpenAI’s Whisper on a MediaTek Genio board for offline speech-to-text, no cloud fallback, no exceptions. Getting Whisper running on the Genio NPU turned out faster and cleaner than anyone on the call expected. Then, a few weeks into integration testing, the board started rebooting itself in the middle of a conversation, and the release sat on hold until we found out why.

Key Insights

• Whisper-medium ran accelerated on the Genio NPU faster than our own bring-up estimate, which surprised us given how little public documentation exists for edge Whisper deployments on this class of NPU.
• We picked Whisper-medium over smaller, cheaper variants because of NPU operator coverage, not because of accuracy or latency targets. Model choice was a compatibility decision first.
• A demo that ran cleanly for five-minute test sessions eventually crashed the AI Processing Unit hard enough to force a full board reboot, not just an application restart.
• The failure was invisible under any short demo and only surfaced after enough continuous inference cycles had run, which is exactly the kind of bug that survives a client walkthrough and fails in the field.
• The standard mitigation for this class of failure has nothing to do with the model itself, it’s about how the model gets compiled for the accelerator.

Whisper on the MediaTek Genio NPU: what we set out to build

The client was building a voice-driven embedded product that had to work with no network connection available, on hardware with real power and thermal limits. They already had Whisper running on the CPU as a proof of concept, and it worked, but it was too slow for anything resembling a natural back-and-forth exchange. Moving speech-to-text onto the NPU was not a nice-to-have. It was the only path to an acceptable response time without moving to a bigger, more expensive SoC that the product’s cost target could not absorb.

Going in, we did not know whether Whisper’s architecture would map cleanly onto Genio’s accelerator hardware. Whisper is an encoder-decoder transformer, heavy on attention and with more dynamic shape behavior than the CNNs most edge NPU tooling is built and tested around. Genio’s AI subsystem splits work across the NPU and MDLA, and it was not obvious ahead of time how much of Whisper’s graph would land on the fast fixed-function path versus fall back to the more general-purpose cores.

We deliberately stayed on the general-purpose inference runtimes for this, ONNX Runtime with the NeuronExecutionProvider as our primary path, rather than reaching for a vendor-specific transformer toolchain that would have tied the whole deployment to a narrower, less field-tested code path. That decision mattered more than it looked like at the time. The standard ONNX and TFLite delegate paths are what MediaTek’s own validation has exercised most heavily, and staying on well-worn tooling meant that whatever came up later, including the crash, there would be existing techniques and reference points to debug it with, instead of fighting an SDK nobody on the team had scar tissue with.

The first bring-up went well. Whisper-medium transcribed test audio fast and accurately on the NPU, and the demo looked good enough that everyone on the call, including us, was a little surprised. Very few teams were running Whisper with NPU acceleration on this class of edge silicon at the time, so there was no playbook to compare notes against. We were writing the playbook as we went.

Why we shipped Whisper-medium instead of a smaller model

The obvious move for an edge device is to pick the smallest model that meets your accuracy bar, since every parameter you shed buys back latency and power headroom. We didn’t do that, and the reason was operator coverage, not accuracy.

Every Whisper size shares the same architecture shape, so the question was never “does the encoder-decoder pattern run on this NPU.” It was “does this specific size, at this specific export configuration, hit only operators the delegate has already proven out.” Smaller models are not automatically safer here. A smaller model still needs every operator in its graph individually validated against the NPU delegate, and a variant nobody has run before carries the same discovery risk as a custom architecture, regardless of its parameter count.

Whisper variant	Approx. parameters	Why we didn’t start here
tiny	~39M	Smallest footprint, but its operator graph had no track record on this delegate path; every layer would need its own validation pass
base	~74M	Same coverage risk as tiny, and the power savings over medium weren’t worth the accuracy we’d be giving up during initial bring-up
small	~244M	A reasonable phase-two target once the delegate path itself was proven stable in production
medium	~769M	Matched the reference model MediaTek’s own NPU tooling had already been validated against, so we shipped this first

The plan from day one was to ship the vendor-validated model, get it stable in production, and only then qualify a lighter variant against real field data. That sequencing is what let us separate two different kinds of risk, model accuracy risk and NPU delegate risk, and debug them one at a time instead of at once, which mattered once the crash showed up and we needed to know whether the model or the delegate path was the moving part.

The demo worked. Then the board started rebooting.

Short test sessions never showed a problem. Five minutes of back-and-forth, everything clean, every transcription on time. The first sign of trouble came out of the client’s own QA process, once someone ran a longer soak session that pushed the device through far more consecutive utterances than any of our demo scripts had. Partway through, the board rebooted. Not the application, the whole board, mid-conversation.

That distinction mattered immediately. An application crash gives you a stack trace and a core dump. A full board reboot with no corresponding application-level exception means the fault happened somewhere our own process couldn’t see, most likely in the AI Processing Unit’s firmware or the kernel driver managing it, not in our inference code. We had to go looking at a different layer than we’re used to debugging at, closer to the accelerator’s own logs than to anything our application had logged.

This is a known failure signature for NPU-accelerated inference under sustained load, and it’s worth knowing the general troubleshooting order before you’re staring at it on a real device. Thermal throttling and memory pressure are the first things worth ruling out, a temperature log and an instrumented memory check are cheap and fast, since they’re the most common causes of hardware-level instability under load. But in workloads that call an NPU delegate over and over for hours at a time, the more common root cause sits somewhere else: how often the accelerator is being asked to recompile.

The usual culprit is the online compilation path. ONNX Runtime’s NeuronExecutionProvider and the TFLite Neuron delegate both compile NPU subgraphs at session start by default, every time a session begins, rather than reusing a fixed, precompiled representation already resident on the accelerator. In a five-minute demo that overhead is invisible, a few milliseconds nobody notices. Under continuous use, hour after hour of back-to-back requests, that repeated compilation is the kind of thing that can accumulate into instability in the delegate’s state or the firmware handling it, and it’s the first hypothesis worth testing when a crash only shows up under sustained load rather than in short sessions.

The standard mitigation is to stop recompiling on every session. Moving a fixed model like Whisper-medium onto the offline, ahead-of-time compilation path instead, compiling once on a host machine into a fixed accelerator binary and loading that same binary on every boot, removes the repeated-compile step entirely. For any NPU-accelerated model that doesn’t change between runs, that’s the first thing we’d check before anything else. Extended soak testing, thousands of consecutive inference calls back to back, is how you’d confirm the fix holds before it ships, not a five-minute demo.

What we’d tell any team doing NPU-accelerated ASR on Genio

A few things we’d do again without hesitation, and one thing we’d do earlier:

1. Ship the vendor-validated model first. Don’t spend your first bring-up cycle chasing the smallest model that meets your accuracy target. Prove the delegate path with something known to work, then optimize model size once that path is stable.
2. Do not trust demo-length testing. A clean five-minute session tells you almost nothing about a workload meant to run for hours. Any accelerator-backed inference path needs a soak test long before it needs a benchmark.
3. Default to offline compilation for any fixed model running continuously. If the model isn’t changing between sessions, there is little reason to pay a compile cost, and apparently some risk, on every single inference call.
4. Budget review time at the accelerator layer, not just the application layer. When a symptom looks like “the board rebooted” instead of “the app crashed,” you’re debugging firmware and driver behavior, and that takes different tools and a different kind of patience than a normal application bug does.

Frequently Asked Questions

Can OpenAI’s Whisper run on an NPU?

Yes. Whisper’s encoder and decoder are standard transformer blocks, convolutions, matrix multiplies, layer norms, and attention, and most of those operators map onto general-purpose NPU inference paths like ONNX Runtime’s NeuronExecutionProvider or a TFLite Neuron delegate. Operator coverage varies by NPU generation and toolchain version, so validate the specific model size against the specific silicon before committing to it in a product.

Which Whisper model size works best on an edge NPU like MediaTek Genio’s?

Start with whichever size the vendor’s own NPU delegate has already been validated against, even if a smaller model would fit your latency budget better on paper. We shipped Whisper-medium first because its operator graph matched what MediaTek’s tooling had already been exercised on, and planned to qualify lighter variants only after the delegate path itself proved stable.

Why would NPU-accelerated inference cause a full device reboot instead of an application crash?

Because the fault happens below your application, in the accelerator’s firmware or driver stack, not in your process. A hard enough fault in the NPU or its supporting co-processor can trigger a watchdog-driven full system reset rather than an isolated, catchable crash. That’s why the symptom shows up as a random reboot with no application-level stack trace to chase.

What is the difference between the online and offline NPU compilation paths on MediaTek Genio?

The online path, used by ONNX Runtime’s NeuronExecutionProvider or a TFLite Neuron delegate, compiles NPU subgraphs at model load time, every time the session starts. The offline path compiles the model ahead of time on a host machine into a fixed hardware binary that the device loads once and reuses. For a model that doesn’t change between runs, the offline path skips repeating that compile step and tends to be the more stable choice under continuous, production load.

Does MediaTek Genio support fully offline speech-to-text with no cloud API?

Yes. Whisper can run entirely on-device on Genio hardware with an NPU, through either ONNX Runtime or TFLite, with no network round trip needed for transcription. The tradeoff is engineering time spent on model size selection, delegate configuration, and validation under sustained load, not a fundamental limitation of the platform.

---

ProventusNova helps hardware startups solve embedded systems problems fast. Need Whisper, or another model, running reliably on Genio’s NPU? See our edge AI deployment service.