Linux networking / receive path

Linux NAPI, IRQs, and the NET_RX softirq

A packet does not normally trigger an interrupt, run through TCP, and arrive at your process in one heroic motion. Modern Linux uses interrupts to announce work, then polls batches of completed DMA descriptors.

A narrow, deep note for building the right mental model before debugging packet loss.

Why one interrupt per packet collapses

At 10 million packets per second, interrupt-per-packet means 10 million hardware interrupts every second. Each entry costs pipeline disruption, register handling, cache pollution, driver work, and scheduling interference. Even a fictional 1 μs per interrupt would demand ten seconds of CPU time per second.

Polling amortises fixed cost. One interrupt can announce a batch, and one poll can process 64 or more completed descriptors. Permanent polling has the opposite failure mode: an idle CPU repeatedly checks an empty ring. NAPI combines both behaviours:

The objects in the path

RX descriptor ring

A NIC and its driver share a circular array of descriptors. Each descriptor points to a DMA-mapped buffer. The driver supplies empty buffers; the NIC writes frames and completion metadata; the driver reclaims completed entries and replenishes them. If replenishment reaches zero, the NIC has nowhere to DMA the next frame. That packet may be dropped before any skb, firewall hook, or socket counter exists.

RXQ3
[ descriptor 0 → buffer A ] [ descriptor 1 → buffer B ] …
       NIC owns posted entries; driver owns completed entries

napi_struct

Each NAPI instance has a driver poll callback. A multi-queue NIC commonly has one RX queue, MSI-X vector, and NAPI instance per queue, allowing independent queues to run on different CPUs. The relationship is common, not an API law.

sk_buff and page pools

An skb is mostly metadata: protocol, device, headers, checksum state, hash, and references to packet memory. Bytes often remain in page-pool-backed memory. Page pools recycle pages in the queue/NAPI context, avoiding an allocate/map/free cycle for every frame. The page cannot return to the RX ring while an skb, XDP frame, socket, or zero-copy consumer still references it.

IRQ → NAPI → NET_RX

1. NIC receives and classifies

RSS hashes packet headers and selects an RX queue through an indirection table. Packets in one flow normally stay on one queue, preserving ordering. The NIC DMA-writes the frame, marks the descriptor done, and may raise an MSI-X interrupt for a batch.

2. Hard IRQ schedules work

irqreturn_t driver_irq(int irq, void *data)
{
    struct queue *q = data;
    if (napi_schedule_prep(&q->napi)) {
        mask_rx_irq(q);
        __napi_schedule(&q->napi);
    }
    return IRQ_HANDLED;
}

The handler stays short. Once NAPI owns the queue, further RX interrupts are normally masked or suppressed; the upcoming poll discovers new arrivals.

3. NET_RX softirq polls

Scheduling NAPI raises networking softirq work. net_rx_action() walks the current CPU's NAPI list and calls each poll method. Processing is bounded by both packet budget and time budget so receive work cannot monopolise the CPU.

4. Driver poll drains the ring

poll(napi, budget):
    reclaim TX completions
    while work_done < budget and RX descriptor is complete:
        build skb around received memory
        napi_gro_receive(skb)
        refill descriptor
        work_done++
    if work_done < budget:
        napi_complete_done(napi, work_done)
        re-enable RX interrupt

Returning exactly the budget usually means more work may exist, so NAPI remains scheduled. Returning less than budget lets the driver complete NAPI and safely re-arm interrupts. Completion versus interrupt re-enable is a race boundary: there must never be a gap where neither scheduled polling nor armed interrupts owns pending work.

GRO changes what you count

Drivers commonly call napi_gro_receive(). GRO merges adjacent packets from one flow before the expensive upper-stack work. Ten wire TCP segments might become one roughly 14 KB GRO skb. Therefore:

After GRO flush, a common path is netif_receive_skb() → ingress TC/netfilter/bridge/VLAN handling → ip_rcv() → routing → TCP or UDP → socket queue → application. XDP, tunnels, namespaces, and offloads can change the exact path. NAPI ends the driver/RX-queue phase; it is not the IP stack.

When ksoftirqd appears

Receive softirq work may run immediately after interrupt handling. If it remains pending too long, Linux defers it to ksoftirqd/N, a schedulable kernel thread. Under contention it can lose CPU time, causing RX rings and backlogs to grow even while aggregate host CPU appears mostly idle.

Budget exhaustion is not automatically loss. It means a pass stopped with work unfinished. Repeated exhaustion means arrival rate may be approaching or exceeding service rate. Raising budgets can postpone drops but can also starve applications, routing daemons, storage clients, and health checks.

Three queues, three failure modes

In /proc/net/softnet_stat, historically important first fields are hexadecimal processed, dropped, and time-squeeze counters. Interpret deltas per CPU, not lifetime totals. Rising time_squeeze says processing repeatedly hit a constraint; it is a saturation warning, not proof of packet loss.

Locality, RSS, and moderation

Inspect /proc/interrupts, /proc/irq/<irq>/smp_affinity_list, ethtool -x eth0, and ethtool -S eth0. Aim for RX queue IRQ → NAPI → protocol processing → application on the same CPU or NUMA node where practical. Manual pinning can fight irqbalance.

RSS distributes flows, not bytes. One elephant flow still maps to one queue. Tunnel traffic can collapse thousands of inner flows onto one outer hash unless hardware hashes inner headers. Interrupt moderation trades fewer interrupts and better throughput for more latency and burst size; tune per workload, not by superstition.

A useful diagnostic order

1. Compare per-CPU %soft, ksoftirqd/N, and application delay.
2. Compare per-queue interrupt deltas for RSS imbalance.
3. Check hardware drops and no-buffer counters.
4. Track per-CPU softnet dropped/time-squeeze deltas.
5. Inspect ring size, channel count, RSS indirection, IRQ affinity, and NUMA placement.
6. Only then test budget, backlog, GRO, and interrupt-moderation changes.

Mental model: wire → NIC parser/RSS → RX queue → DMA descriptors → MSI-X IRQ → IRQ CPU → NAPI poll → GRO → NET_RX budget → RPS/backlog → IP/TCP/UDP → socket queue → application. Every arrow can queue, drop, migrate CPUs, or lose locality.