When camera frames move between two boards, the link speed is rarely the problem. The problem is who moves the frame. If the CPU copies the data byte by byte, the PCIe link runs at a fraction of its capacity and that core can do nothing else. When the job goes to the DMA engine inside the controller, the CPU only says "from here to there, this much", and the hardware does the rest. On our bench, between a TI Jacinto 7 (J784S4) and a Jetson AGX Orin over PCIe Gen3 x4, we measured 3.0–3.2 GB/s with DMA; on the same link, DMA ran about 8 times faster than CPU-driven copies. This article explains where that gap comes from and which settings decide it.

In short

  • If the CPU copies the frame, the bottleneck is the CPU, not the link. With DMA the CPU only describes the transfer; the controller moves the data.
  • The raw ceiling of PCIe Gen3 x4 is 3.94 GB/s per direction. Packet headers take a share of it; raising the maximum payload size from 128 to 256 bytes lifts packet efficiency from about 86% to 92%.
  • DMA has a fixed setup cost. For transfers of a few kilobytes a CPU copy can win, which is why deriving "time per frame" from bandwidth is misleading.
  • What we measured: 3.0–3.2 GB/s and about 8 times. What we did not: end-to-end latency of the four-Orin line.

The film and the work behind it

Our 64-second film Straight to the GPU follows one frame: it lands on the Jacinto, crosses PCIe into the Orin's memory, four Orins in turn do video stabilization, image enhancement, object tracking and gauge reading, and then the frame goes back to the Jacinto and out over WebRTC to the operator's screen. Then the same boards and the same cable are shown with the CPU copying every frame.

The factory, the four-Orin line and the control room are 3D illustrations, and the data paths and detections are animated. What is real is the driver work between two boards and its bench measurement: a Jacinto 7 (J784S4) on the root side, a Jetson AGX Orin on the endpoint side, PCIe Gen3 x4 between them. Running the Orin as an endpoint is a mode NVIDIA documents. We write the driver on both sides. We have not built and measured a four-Orin line.

Why is a frame copied, and who does the copying?

When two boards are connected over PCIe, the endpoint exposes part of its memory to the root side as an address window (a BAR). The driver on the root side maps that window into its own address space, and the simplest transfer is a memcpy into the mapping. It works, but every byte goes through the CPU: the CPU reads the data and writes it into the window, and the controller turns that into packets (TLPs) one after another. The core is busy for as long as the copy runs, and the speed is the speed of that loop, not of the link.

With DMA the division of labour changes. The CPU gives the controller a source address, a destination address and a length; the DMA engine inside the controller takes the data from memory and drives it onto the link. In our setup that engine is on the endpoint, in the Orin's PCIe controller. The Linux kernel carries a driver for this kind of controller (dw-edma; its Kconfig text says "normally implemented on endpoints SoCs"). The PCIe controller on the Jacinto side comes from a different IP family (Cadence); assuming the two sides are the same hardware is the first trap.

The kernel documentation states the difference plainly. The NVMe endpoint driver's documentation says that if DMA is not supported, MMIO is used, "which results in poor performance". The cover letter of the same driver reports 2.82 GB/s for 512 KB random reads on an RK3588 board with DMA. Our measurement is on different boards and a different workload; we mention it to show that the order of magnitude is not ours alone, not to put the numbers side by side.

PCIe 3.0 carries 8 GT/s per lane and uses 128b/130b encoding. For four lanes:

4 lanes × 8 Gbit/s × 128/130 ÷ 8 = 3.94 GB/s      (per direction, raw)

No transfer reaches that ceiling, because the data is packetised. Every packet has a header, a sequence number and error checking; Xilinx's WP350 note puts that overhead at 20 to 30 bytes. The most data one packet may carry (Max_Payload_Size) is a value the two ends agree on: it is defined from 128 to 4096 bytes, and every device in the hierarchy must use the same value.

Max payload Packet efficiency (WP350, 20 bytes overhead) What is left of 3.94 GB/s
128 bytes 86% about 3.4 GB/s
256 bytes 92% about 3.6 GB/s
512 bytes 96% about 3.8 GB/s

The last column is our multiplication, not a measurement; Gen3 framing adds slightly more than 20 bytes, and flow-control packets use the link too. Our boards negotiated 128 bytes at boot; both support 256, so we raised it. The 3.0–3.2 GB/s we measured is 76–81% of the raw ceiling.

There is also a unit trap. Some test tools print 2²⁰ bytes as "MB". At the GB scale the difference is 7% (2³⁰ / 10⁹ = 1.074); do not compare two measurements without knowing which unit the tool prints.

The price of DMA: small transfers

DMA is not free. Each transfer needs a command to be prepared, addresses to be written, and an interrupt when it completes. That setup is a fixed time, and its share of the total grows as the transfer shrinks. In our measurements a CPU copy can beat DMA for transfers of a few kilobytes; at megabytes, DMA is clearly faster.

Two practical consequences follow. The first is a division of labour: control flags and small parameters go by CPU, large payloads such as frames go by DMA. The second is honesty: "3 GB/s means a 3 MB frame takes 1 ms" is wrong, because it ignores the fixed setup time. Time per frame has to be found by measuring a transfer of that size, not by dividing by the bandwidth. That is why the film shows no milliseconds.

Memory type: cached or write-combining?

How the shared memory is mapped decides the local access speed. The kernel's device I/O documentation describes the two extremes: ioremap() bypasses the cache and issues every access to the device separately; ioremap_wc() allows writes to be combined and, in the words of the documentation, is typically used for video framebuffers, where it can increase the performance of writes. Normal cached memory is far faster for reading and comparing, on one condition: when the other side writes to that memory by DMA, the CPU cache must not be holding the old data.

If the hardware guarantees coherency (the controller sits on an interconnect that is coherent with the CPU caches), a cached mapping is safe. If it cannot, you either flush the cache on every transfer, which costs CPU time, or you stay with write-combining. We made different choices on the two sides: cached where we could verify coherency, write-combining on the other. The rule is simple: correctness is tested first (random data, random addresses, compare), and speed is measured after that.

Who says "the frame has landed"?

An endpoint notifying the root side is standard: the kernel's endpoint framework offers pci_epc_raise_irq() for it and raises MSI or MSI-X. The reverse direction is different. PCIe has no ready-made interrupt path for the root side to tell an endpoint "there is new work"; Linux only gained "doorbell" support in that direction in version 6.17, and only under certain hardware conditions. Even the in-tree NVMe endpoint driver works by constantly polling its queues.

Polling spends one of two things: a core, or reaction time. We use interrupts in both directions; that is the basis for the line in the film, "each one starts the moment the frame lands". We have not measured how this scales on a line with four units.

What does "straight into GPU memory" mean?

Zero-copy does not mean the data never moves: the frame crosses the link once. It means that before and after that crossing the CPU does not copy the frame from one buffer to another. Two things make that possible on the Orin. The memory reserved for the transfer is physically fixed and contiguous, so that units such as the GPU can reach the same region. And, as NVIDIA's "CUDA for Tegra" note says, on Tegra device memory and host memory are allocated on the same physical DRAM, so duplicate allocations and data transfers can be avoided.

This article does not go into how that region is mapped on the GPU side. We describe the rest of the inference pipeline on our NVIDIA Jetson page and the Jacinto side on our Texas Instruments page.

The last step to the operator

After processing, the frame goes to the operator's screen over WebRTC, the protocol family defined by the W3C and the IETF for real-time media in the browser. The delay of that step is independent of PCIe: it is decided by encoding, the network and the browser's buffer. We describe the method in our low-latency video note; we give no measurement here.

Where it goes wrong

  1. Taking bandwidth for latency. GB/s on a large transfer does not tell you the time of a small one; the fixed setup time dominates.
  2. Staying at the default packet size. Two ends that support a larger value may still negotiate a small one at boot. Read the negotiated value.
  3. Fast but stale data. A cached mapping looks fast in a compare test; if it does not see what the other side wrote, the result is wrong. Test data must be random, and memory must be cleared on every round.
  4. Reserved memory regions that collide. Transfer memory is reserved by hand in the device tree. If it overlaps another unit's region, the kernel only warns at boot; after that it is silent data corruption.
  5. Assuming DMA is always fast and always available. For small transfers the fixed setup time dominates, and not every DMA engine supports every operation: the kernel's own test guide says the DMA copy test will fail on most DMA-capable endpoint controllers. Measure the ratio on your own board.

What this article cannot tell you

We measured two things: 3.0–3.2 GB/s between two boards with DMA, and about 8 times the speed of CPU copies for 128 MB transfers. We did not record separately which of the two speeds belongs to which direction, so we give a range. The 8-times ratio comes from an earlier measurement round and has not been re-measured at the current DMA speed.

What we have not measured: end-to-end latency of the four-Orin line, the time for one frame from camera to operator, interrupt latency under load, and stability over long runs. To answer those we would measure:

  • Transfer time at frame size: not the average but p99 and the worst case, with real frame sizes.
  • Time from interrupt to work: from the moment the frame lands to the moment the GPU job starts, with hardware timestamps.
  • The sum of the chain: how the delay grows as units are added.
  • Glass-to-glass latency: from the event the camera sees to the pixel on the operator's screen.

Building this kind of line on your own boards is the driver layer of an edge AI pipeline.

Frequently asked questions

What does zero-copy PCIe mean?

It means the frame crosses the link between two boards once, and the CPU does not copy it from one buffer to another before or after that. The controller's DMA engine moves the data; the CPU only describes the transfer.

The raw ceiling is 3.94 GB/s per direction (4 lanes × 8 Gbit/s × 128/130). Packet headers and flow control take part of it; the loss shrinks as the maximum payload size grows. On our bench we measured 3.0–3.2 GB/s with DMA.

Is DMA always faster than a CPU copy?

No. DMA has a fixed setup cost on every transfer, so a CPU copy can win for transfers of a few kilobytes. At megabytes DMA is clearly faster: about 8 times for 128 MB transfers on our link.

Can latency per frame be calculated from bandwidth?

It should not be. The division ignores the fixed setup time, the interrupt latency and the other steps in the chain. A transfer at frame size has to be measured directly.

Sources

Accessed: 1 October 2026.