Running inference on MediaTek Genio: NeuroPilot, TFLite, and ONNX

Running inference on MediaTek Genio’s APU requires INT8 quantized models and the NeuroPilot TFLite delegate — a different stack than TensorRT on Jetson or standard TFLite on CPU. The performance you get depends on how well your model architecture maps to the APU’s supported operator set. This post covers the full deployment workflow from model conversion to running on-device.

Key Insights

• NeuroPilot SDK provides a TFLite delegate that routes supported ops to the Genio APU — no separate compilation step required for basic deployment
• INT8 quantization is required for APU execution — float32 models run on CPU by default
• ONNX → TFLite conversion is a required step before using NeuroPilot; there is no direct ONNX-to-APU path
• Unsupported ops fall back to CPU automatically, which can negate APU performance gains on models with many unsupported layers
• Latency scales with model complexity non-linearly — benchmark on hardware, not TOPS estimates

How Genio’s APU inference stack works

The MediaTek Genio APU is a dedicated neural network accelerator. It is not a GPU and has no general compute capability. It runs a fixed set of operations at INT8 precision, very efficiently, and routes anything it cannot handle to the CPU automatically.

The software stack is NeuroPilot SDK, which implements a TFLite delegate — a plug-in mechanism that lets TFLite route specific ops to hardware accelerators while keeping the rest on CPU. From the application’s perspective, inference still happens through the TFLite runtime. The delegate handles dispatching operations to the APU transparently.

This design has two implications. First, deployment is simpler than platforms that require a full offline compilation step (like TensorRT on Jetson). You do not need to compile the model to a hardware-specific binary at deployment time — the delegate does this at load time. Second, operator compatibility becomes important. If your model has a high percentage of unsupported operations, those run on CPU, and the APU only handles the supported subset.

Model requirements for APU execution

Before the APU runs your model, two requirements must be met:

INT8 quantization. The APU does not execute float32. A float32 TFLite model loaded with the NeuroPilot delegate runs on CPU — you get NeuroPilot overhead with no APU benefit. Quantize your model before deployment.

Two quantization paths:

1. Post-training quantization (PTQ) — convert an existing float32 model. Requires a representative dataset for calibration. Fast, but accuracy may drop for some model types.
2. Quantization-aware training (QAT) — train the model with simulated quantization. Better accuracy preservation, requires retraining.

For detection models (YOLO, EfficientDet), QAT typically preserves accuracy better than PTQ. For classification (MobileNet, EfficientNet), PTQ usually works well.

Supported operator set. Common CNN operators are supported. Transformer and LSTM-heavy architectures are partially or not supported at the APU level. Check the NeuroPilot SDK documentation for the current operator list — it changes between SDK versions.

Deployment workflow

From TensorFlow / Keras:

# Post-training INT8 quantization
import tensorflow as tf

def representative_dataset():
    for image in calibration_images:
        yield [image[np.newaxis, :].astype(np.float32)]

converter = tf.lite.TFLiteConverter.from_saved_model("saved_model_dir")
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8

tflite_model = converter.convert()
with open("model_int8.tflite", "wb") as f:
    f.write(tflite_model)

From ONNX:

# Convert ONNX → TFLite (using onnx2tf)
pip install onnx2tf
onnx2tf -i model.onnx -o ./tflite_output

# Then quantize the exported TFLite model using the PTQ path above

Running inference with NeuroPilot delegate:

import tflite_runtime.interpreter as tflite

# Load the NeuroPilot delegate
delegate = tflite.load_delegate("libneuropilot.so")

interpreter = tflite.Interpreter(
    model_path="model_int8.tflite",
    experimental_delegates=[delegate]
)
interpreter.allocate_tensors()

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

interpreter.set_tensor(input_details[0]['index'], input_data)
interpreter.invoke()
result = interpreter.get_tensor(output_details[0]['index'])

The libneuropilot.so path depends on your Yocto build configuration — check the SDK sysroot for the exact location.

Benchmarking and verifying APU utilization

The NeuroPilot SDK includes a benchmark tool. Use it to verify that your model is actually running on the APU (not falling back to CPU):

# Run the benchmark tool
neuropilot_benchmark \
  --model model_int8.tflite \
  --num_runs 50 \
  --report_delegate_profiling true

The output shows per-layer execution location (APU vs CPU) and latency. If most layers are showing as CPU in the profiling output, check your quantization — the model may be running as float32.

Common causes of unexpected CPU fallback:

• Model not fully quantized (mixed float32/int8 layers)
• Operator not in the supported list (the benchmark output names which ops are unsupported)
• Tensor shape not supported by APU (very small or unusual shapes may fall back)

Latency expectations by model class

These are representative figures on Genio 700 APU. Treat them as planning estimates — benchmark your actual model.

Model	Quantization	Estimated APU latency
MobileNetV3-Small	INT8	5–8 ms
MobileNetV2 (224×224)	INT8	8–12 ms
EfficientNet-Lite0	INT8	10–15 ms
EfficientDet-Lite0	INT8	12–18 ms
YOLOv5n	INT8	20–30 ms
EfficientDet-Lite2	INT8	30–50 ms

For models with significant unsupported op content (transformers, complex attention mechanisms), the effective latency is the APU latency for supported ops plus CPU latency for unsupported ops. Models designed specifically for efficient deployment on mobile/edge APUs (MobileNet family, EfficientNet-Lite family) tend to have the best APU utilization.

For module comparison on how APU TOPS affects model selection, see Genio 510 vs 700 vs 1200: which MediaTek module for your product.

MediaTek’s NeuroPilot developer portal is at iot.mediatek.com. The TFLite documentation for delegates is at tensorflow.org.