Linux memory and networking

DMA, RDMA, and the memory behind a packet

DMA is ordinary NIC plumbing: a device moves bytes between itself and local RAM without asking the CPU to copy every byte. RDMA uses that capability across machines, with an RNIC writing into registered remote memory.

First principle: DMA and RDMA are related, not interchangeable.

DMA solves a local copying problem

With programmed I/O, a CPU would repeatedly read device registers and store bytes into RAM. At modern packet rates, those loads, stores, loops, PCIe transactions, and cache effects consume the machine before applications get useful work done.

ordinary NIC receive wire → NIC → PCIe DMA write → RAM buffer ↑ CPU prepares destination; NIC moves payload

Linux still uses the CPU for control work: allocate or prepare memory, create descriptors, tell the NIC where the ring lives, ring a doorbell, and process completions. “Direct” means the CPU does not execute an instruction for every payload byte.

Three addresses, never one

Address	Used by	Meaning
CPU virtual address	Kernel code	Address dereferenced through CPU page tables.
Physical address	Memory system	Location of RAM after CPU translation.
DMA address	Device	Address placed in a PCIe transaction; may be translated by an IOMMU.

A driver must not assume virt_to_phys() gives a device-safe address. DMA address may be restricted by the device's DMA mask, translated through an IOMMU, or affected by bus topology.

NIC-visible DMA address ↓ IOMMU / bus translation ↓ physical RAM page

Linux DMA API and ownership

The DMA API hides architecture differences: address width, IOMMU mappings, cache coherence, alignment, and bounce buffers. Common interfaces include dma_map_single(), dma_map_page(), dma_unmap_*, dma_alloc_coherent(), and explicit sync helpers.

Direction is named from the device's perspective:

Traffic	Direction
NIC writes received packet into RAM	`DMA_FROM_DEVICE`
NIC reads outgoing packet from RAM	`DMA_TO_DEVICE`
Both sides may update buffer	`DMA_BIDIRECTIONAL`

Coherent versus streaming

dma_alloc_coherent() is common for descriptor and control memory. Coherence does not remove ordering requirements: a device can observe an ownership bit before address and length unless the driver uses the right barrier.

Streaming mappings fit packet buffers. Ownership moves in a lifecycle:

CPU owns buffer → dma_map_* → device owns buffer device completes → sync/unmap → CPU owns buffer again

RX ring: device owns bytes, driver owns state

A receive descriptor contains a DMA address, length, and ownership/status fields. During setup, Linux allocates a page, maps it with DMA_FROM_DEVICE, stores the DMA address in the descriptor, and gives the entry to the NIC. The NIC reads the descriptor, writes the frame, then marks completion. The driver later builds an skb around the page rather than copying payload into a second buffer.

page = alloc_page(GFP_KERNEL);
dma = dma_map_page(dev, page, 0, PAGE_SIZE, DMA_FROM_DEVICE);
if (dma_mapping_error(dev, dma))
    return -ENOMEM;
rx_desc[i].pkt_addr = cpu_to_le64(dma);
/* memory barrier before ownership is handed to device */

DMA is a state machine, not magic:

free → CPU-owned → posted/device-owned → completed/CPU-owned → recycle

Recycling a page while an skb, socket, XDP frame, or application still references it lets the NIC overwrite data still being read. Freeing memory before stopping in-flight DMA can corrupt an unrelated subsystem.

Cache visibility and ordering

Cache coherence and memory ordering are separate. On coherent x86 servers, stale-cache bugs may hide. Linux also supports non-coherent ARM and embedded systems, where a CPU may need dma_sync_single_for_cpu() before reading device-written data and dma_sync_single_for_device() before handing it back.

Ordering matters even when all agents eventually see the same bytes. A driver may write address and length, execute dma_wmb(), then publish the ownership bit. Without that protocol, hardware can see “ready” before the descriptor fields are ready.

Page pools avoid mapping churn

Allocating, mapping, unmapping, and freeing a page per packet is expensive, especially with an IOMMU. A page pool keeps pages associated with an RX queue/NAPI context, often mapped for reuse:

page pool allocates page ↓ DMA-mapped buffer stays with queue NIC writes frame → stack consumes reference ↓ last reference returns page pool recycles page → RX ring

The optimisation depends on correct reference ownership. Pool statistics showing many in-flight pages can indicate slow consumers, a leak, or simply a workload holding socket buffers longer than expected.

IOMMU: isolation and translation

Without an IOMMU, a DMA-capable device is powerful enough to read or overwrite host memory. An IOMMU maps device-visible addresses into permitted physical pages, enabling VM assignment and protecting other memory. It can also provide a convenient device address space over physically scattered pages.

Costs include IOTLB misses, page-table walks, mapping updates, and invalidation. High-performance drivers therefore keep RX pages mapped and recycle them. Check enablement and faults with:

lspci -vv -s 0000:3b:00.0   # look for BusMaster+
dmesg | grep -Ei 'DMAR|IOMMU|AMD-Vi|fault'
cat /sys/class/net/eth0/device/numa_node
ethtool -g eth0
ethtool -S eth0 | grep -Ei 'dma|no.?buf|miss|alloc|page'

Legacy or constrained devices may need a bounce buffer: NIC DMA-writes into low memory, then software copies into the original high-memory buffer. Functionality survives; the copy tax returns.

Cloud NICs still use the model

A virtual NIC does not mean “no DMA.” From a guest's point of view, a driver still posts RX buffers and a device still DMA-writes guest memory. Hypervisor or provider hardware maps that activity safely. Google's GVE driver, for example, documents fixed-size buffers placed on a hardware RX queue and packets that may span descriptors. The virtual boundary changes implementation, not the ownership model.

Reset safety is where bugs become corruption

Normal receive code assumes a trusted queue state. Reset code runs when hardware may be wedged. Safe teardown must stop queues, disable device DMA, wait for in-flight writes, unmap buffers, and only then free memory. Doing it in the opposite order can yield IOMMU faults, page corruption, random slab damage, or an unrelated process crash.

Important distinction: “NIC reset completed” is not automatically proof that every old DMA transaction has stopped.

DMA versus RDMA

Term	Scope	Example
DMA	Device ↔ local host memory	NIC writes Ethernet frame into a kernel RX page.
RDMA	Host A memory ↔ Host B memory	RNIC writes into a registered buffer on another machine.

For ordinary TCP:

wire → local NIC DMA → kernel RX buffer / skb → TCP stack → socket queue → copy_to_user() → application buffer

For an RDMA write:

application buffer A → RNIC DMA-read → network → remote RNIC DMA-write → registered application buffer B

RDMA adds remote addressing, memory registration, protection keys, queue pairs, transport semantics, and completion events. CPUs still create queue pairs, register memory, post work requests, ring doorbells, and handle completions. They simply do not copy every payload byte through the normal socket path.

One sentence: DMA means “move bytes between device and my local RAM.” RDMA means “use RNICs to move bytes directly between registered RAM on two machines.”