Edge AI & Computer Vision
Engineering ApproachInference Pipeline Architecture on a Constrained Edge Device
INT8 quantization · CUDA graphs · DLA co-execution · NVIDIA Jetson Orin NX
NVIDIA Jetson Orin NX 16GB · TensorRT 8.6 · JetPack 6
01Problem
Problem: The Inference Budget on a SWaP-C Platform
On an autonomous aerial system, stock FP32 ONNX runtime inference does not produce the throughput required for real-time detection and tracking. On a size, weight, power and cost constrained platform speed is not the only issue: the power budget bounds flight time directly. The target metric is therefore not fps but frames per second per watt.
- 01Insufficient inference throughput causes dropped frames and lost tracks on fast-moving objects
- 02FP32 inference saturates the GPU and leaves no headroom for concurrent image pre-processing
- 03High power draw pushes the platform below its target flight time
- 04Recompiling the model from ONNX at every start delays the operation
- 05With CPU-GPU copies and kernel launch overhead unmeasured, the location of the bottleneck is unknown
02System Context
03Root Cause Analysis
- 01The model ran in FP32, the Orin NX's INT8 Tensor Cores and DLA capacity sat completely idle
- 02ONNX Runtime used generic execution; TensorRT's layer fusion and kernel auto-tuning were not in play
- 03A separate kernel launch chain per frame generated CPU overhead. GPU stalls were visible in profiling
- 04The engine was rebuilt on every startup; serialization was not used
- 05The entire load sat on the GPU, the DLA cores went unused, leaving power efficiency on the table
04What We Changed
Apply INT8 quantization calibrated with a representative dataset
→ Tensor Core and DLA capacity becomes usable; accuracy loss is validated against the mission set
Enable TensorRT engine serialization
→ Recompilation at startup disappears and model load time becomes predictable
Collapse the per-frame kernel launch chain into a single CUDA graph
→ CPU launch overhead drops and the GPU stays fed
DLA co-execution: suitable layers to the DLA, the rest to the GPU
→ The same work runs within a lower power budget and GPU headroom opens up
Move pre-processing to the GPU (CUDA resize and normalize), reduce CPU-GPU copies
→ The bottleneck moves to the model itself and end-to-end throughput becomes sustainable
05Benchmark Results
06Why It Mattered
fps on its own means nothing; on a SWaP-C constrained platform the meaningful metric is fps per watt. Longer mission time and more concurrent perception work on the same battery both come from improving that ratio.
- 01Real-time multi-object detection and tracking becomes possible on a single device
- 02Lower power draw extends battery life and mission time directly
- 03Freed GPU capacity becomes available for additional perception work such as sensor fusion
- 04The INT8 calibration and validation flow is built as a reusable pipeline; model updates take the same path
To evaluate your Edge AI & Computer Vision needs on your own platform, schedule an embedded architecture audit or scope your platform class with the system requirements calculator.
Methodology Note
- fps is reported over the end-to-end pipeline (capture, detect, output), not as an isolated model benchmark
- Power is measured with external equipment under sustained load after thermal settling
- INT8 accuracy loss is measured against the FP32 reference on a mission-representative validation set
- Comparisons are made on identical hardware, identical input stream and identical power mode
- This page describes a method. When a measurement is published, its setup and method sit next to the number
Is your Edge AI pipeline missing its fps or power budget?
Free technical assessment, let's review the latency and determinism challenges in your system together.
This page describes a method, not a measurement report. Spikedge publishes only measurements taken on its own bench whose setup can be published; when a publishable measurement for this topic exists, it will be added here.