Real-Time Perception on the Edge: Detection, Segmentation, and Depth

Most computer vision demos you see are running in a data center. This one is not. The video below is a modular perception pipeline running entirely on a local machine, served to a browser over the network, with no cloud round-trip. It runs three perception tasks at once, open-vocabulary object detection, instance segmentation, and monocular depth estimation, and reports live metrics for each. This is the kind of system we build for hardware teams deploying vision on NVIDIA Jetson and MediaTek Genio.

Key Insights

• The pipeline runs open-vocabulary detection: the target classes are typed in as text (a person, a laptop, a chair, a desk) and applied live, with no retraining. Change the prompt, and the detector localizes the new objects.
• Three perception tasks run as toggleable stages: detection, instance segmentation, and depth estimation. Each can be turned on or off so you can see, and measure, the compute cost of each stage independently.
• Everything runs locally. The dashboard is served over the LAN, frames never leave the device, and there is no cloud dependency, no round-trip latency, and no data leaving the premises.
• The interface accepts multiple input sources: live webcam, video file, still image, and RTSP network stream, so the same pipeline can be driven by a real camera or a recorded clip.
• A live metrics panel reports FPS, detection count, and per-stage latency, which is exactly the instrumentation you need to tune a pipeline for a real-time target on constrained hardware.

What the Demo Shows

The interface is organized as a control panel and a set of live views.

Open-vocabulary detection. Instead of a fixed class list baked into the model, the active classes are a text prompt you edit at runtime. In the demo, the prompt is a short list, a person, a laptop, a chair, a desk, and the detector draws labeled boxes for exactly those objects. Swap the prompt for warehouse objects, PPE, or defects on a line, and the same deployed model reports the new set. This is the difference between “the model knows 80 things forever” and “the model detects what you ask it to.”

Instance segmentation. Beyond a bounding box, the segmentation stage returns a per-object mask, so a person, a plant, a laptop, and a chair each get a distinct colored silhouette. Masks are what you need when a box is too coarse, for grasping, measurement, occlusion reasoning, or clean overlays.

Depth estimation. The depth view is a single-camera (monocular) depth map, rendered as a near-to-far colormap. No stereo rig, no depth sensor, just an inference model estimating relative distance from a single RGB frame. For robots and drones, a monocular depth stage is a cheap way to add spatial awareness without new hardware.

Live metrics. FPS, detection count, and detection latency in milliseconds are shown as the pipeline runs. This is the part that matters for edge deployment: perception on the edge is a budget problem, and you cannot tune what you do not measure.

Why Run Perception on the Edge at All

Running these three tasks on-device, rather than shipping frames to a server, is a deliberate architectural choice with concrete benefits:

• Latency. There is no network round-trip. For a robot arm, an autonomous mobile robot, or an inspection station, the control loop cannot wait on a server in another building.
• Privacy and data residency. Camera frames never leave the device. For healthcare, retail, and industrial sites, that removes a whole category of compliance and security concern.
• Reliability. The system keeps working when connectivity drops. An edge perception node does not go blind because the uplink went down.
• Cost. No per-frame inference bill, no bandwidth to stream raw video off-site.

The tradeoff is that you are now solving a hard engineering problem: three models, competing for the same GPU, DLA, memory bandwidth, and thermal budget, all while holding a real-time frame rate.

How This Maps to Production Hardware

The demo runs the pipeline in a browser for clarity, but the same architecture is what ships on an embedded module. In production, the pieces map like this:

• Camera ingest comes from CSI or GMSL2 cameras, or an RTSP feed, decoded through a hardware-accelerated GStreamer pipeline rather than a webcam grab.
• Detection and segmentation run as optimized TensorRT engines on the GPU, with layers mapped to the DLA where they are supported to free the GPU.
• Depth runs as a separate engine, scheduled so it does not starve the detector.
• Zero-copy memory keeps frames in GPU-accessible buffers so they are not copied between stages, which is often the difference between hitting and missing a frame-rate target.
• INT8 quantization is applied where accuracy allows, to cut latency and power.

That is the work: taking a pipeline that runs, and making it run in real time, within a power and thermal envelope, on the specific silicon a product ships on. It is exactly the class of problem ProventusNova solves on NVIDIA Jetson and MediaTek Genio, from camera bring-up through to a deployed, profiled inference pipeline.

Building Something Like This

If you are putting cameras and models on an embedded device and need real-time perception that runs locally, the hard parts are rarely the models themselves. They are the camera bring-up, the GStreamer pipeline, the memory path, the accelerator mapping, and the profiling that gets you from “it runs” to “it runs at frame rate, every time.” That is the work we do on a fixed-bid basis.