Real-time video on MediaTek Genio: GStreamer pipeline setup

GStreamer on MediaTek Genio uses a different set of hardware-accelerated elements than Jetson, and the documentation is sparser. Getting real-time video working requires knowing which elements use the hardware codec, how to avoid the system memory bottleneck between pipeline stages, and what the known limitations are. This post covers the full pipeline setup.

Key Insights

• MediaTek provides hardware-accelerated GStreamer elements for H.264/H.265 encode and decode — these work, but documentation is sparse
• Camera input uses standard V4L2 (v4l2src) — not a proprietary element, which is both a strength and a limitation
• No NVMM-style zero-copy memory path: frames move through system RAM between pipeline stages
• Multi-process GStreamer pipelines compete for shared codec hardware — keep inference and video pipelines in the same process
• 4K60 encode is achievable on Genio 700 hardware; 4K120 on Genio 1200

The GStreamer stack on Genio

MediaTek’s IoT Yocto SDK includes a GStreamer plugin package (gst-plugin-mtk or similar depending on SDK version) that exposes the hardware video codec as GStreamer elements. The stack looks like this:

Camera → V4L2 kernel driver → v4l2src → [pipeline] → mtkh264enc → filesink / network
                                                   ↓
                                         waylandsink / kmssink (display)

This is different from Jetson’s architecture, where camera input goes through the Argus API (Jetson-specific ISP) and frames live in NVMM memory from capture through encode. On Genio, V4L2 is the camera interface at the GStreamer level — standard Linux, which is actually easier to work with when you are debugging, but lacks the zero-copy optimization.

Hardware-accelerated elements

The MediaTek GStreamer plugin provides these elements (verify against your SDK version — element names can vary between IoT SDK releases):

Element	Function
mtkh264enc	H.264 hardware encode
mtkh264dec	H.264 hardware decode
mtkh265enc	H.265/HEVC hardware encode
mtkh265dec	H.265/HEVC hardware decode
mtkjpegenc	JPEG hardware encode
mtkjpegdec	JPEG hardware decode

Display sinks: waylandsink (Wayland compositor, the default on Genio Yocto builds), kmssink (KMS/DRM direct, for headless output to display without compositor).

Working pipeline examples

Basic camera preview:

gst-launch-1.0 \
  v4l2src device=/dev/video0 ! \
  video/x-raw,width=1920,height=1080,framerate=30/1 ! \
  waylandsink

Camera to H.264 file:

gst-launch-1.0 \
  v4l2src device=/dev/video0 ! \
  video/x-raw,width=1920,height=1080,framerate=30/1 ! \
  mtkh264enc ! \
  h264parse ! \
  mp4mux ! \
  filesink location=output.mp4

H.264 playback:

gst-launch-1.0 \
  filesrc location=input.mp4 ! \
  qtdemux ! \
  h264parse ! \
  mtkh264dec ! \
  waylandsink

Camera + inference pipeline (single process):

gst-launch-1.0 \
  v4l2src device=/dev/video0 ! \
  video/x-raw,width=640,height=480,framerate=30/1 ! \
  tee name=t \
    t. ! queue ! appsink name=inference_sink \
    t. ! queue ! mtkh264enc ! h264parse ! mp4mux ! filesink location=record.mp4

Known limitations to plan around

No zero-copy memory path. On Jetson, NVMM buffers let frames move from NVCSI through VI through nvvidconv through the encoder without touching CPU-accessible memory. Genio does not have an equivalent. Frames pass through system memory at each GStreamer element boundary. For 1080p30 this is fine. For 4K30 at high data rates, profile your memory bandwidth to make sure it is not a bottleneck.

Multi-process codec contention. The MediaTek hardware codec is a shared resource. If you run your inference pipeline in one process and your video record pipeline in another, both compete for the same encoder/decoder hardware. This is the same problem nvvidconv hits on Jetson with multiple processes — the solution is the same: keep related pipelines in the same process and use tee to branch the stream. Separate processes will work but will perform worse than a unified pipeline.

Sparse error messages. When a mtkh264enc pipeline fails, the error messages are not always specific about why. Missing format capability, unsupported resolution, or hardware resource exhaustion can all produce similar generic errors. Start with a known-good pipeline at a standard resolution (1080p30) and change one variable at a time.

ISP controls. On Jetson, nvarguscamerasrc exposes ISP settings (exposure, white balance, gain) as GStreamer properties. On Genio, ISP control goes through V4L2 controls — v4l2-ctl --set-ctrl. This is more manual but also more standard — the same V4L2 control interface works on any Linux platform.

Real-time video with inference

Running inference alongside video encode on Genio requires keeping everything in one GStreamer pipeline (or using shared memory between processes). A practical split: the encoder hardware runs independently of the APU, so you can record to file while running inference on the APU simultaneously without one consuming the other’s budget.

What competes: CPU cycles (for the GStreamer pipeline thread, element processing) and memory bandwidth (camera frames flowing through system RAM). On Genio 700, a 1080p30 record + 640×480 inference workload is feasible. At 4K30 record + 1080p inference, profile memory bandwidth carefully.

For more on running inference on the APU alongside other workloads, see running inference offline on MediaTek Genio: NeuroPilot, TFLite, and ONNX support.

GStreamer documentation for V4L2 source element: gstreamer.freedesktop.org. MediaTek’s IoT developer portal has the Genio SDK download with the GStreamer plugin package.