Virtualized networking

SR-IOV: PFs, VFs, and Kubernetes

One physical NIC can expose many hardware-backed PCI functions. SR-IOV gives workloads direct queue access while keeping them inside one shared physical failure domain.

The short version: PF manages, VF carries traffic, NIC switches, IOMMU isolates DMA.

Why SR-IOV exists

Traditional VM networking places software between guest and wire: virtio, vhost, TAP, bridge or Open vSwitch, host stack, and physical driver. That path is flexible—host can filter, account, route, mirror, and encapsulate packets—but every software boundary costs CPU, latency, memory operations, and scheduling work.

traditional: VM → virtio → host virtual switch → host NIC → wire +SR-IOV: VM → VF queue → NIC hardware → wire

SR-IOV was standardised as a PCI Express capability around the late 2000s, when 10/40/100 GbE made per-packet software switching increasingly expensive. It exposes hardware resources as multiple PCI functions instead of emulating many NICs in software. “Single Root” means these functions live beneath one PCIe root complex in one host.

PF and VF

Function	Role
Physical Function (PF)	Full-featured PCI function. Owns port, firmware, global resources, VF creation, policy, and often mailbox handling.
Virtual Function (VF)	Lightweight PCI function with its own config space, queues, DMA resources, and MSI-X interrupts, backed by shared NIC silicon.

A VF is a real PCI device, not a veth. It still shares cable, port bandwidth, packet buffers, firmware, embedded switch, PCIe link, and reset domain with its PF and sibling VFs. The PF is administrator; VFs are tenant datapaths.

Enable VFs in Linux

lspci -Dnn | grep -i ethernet
cat /sys/bus/pci/devices/0000:3b:00.0/sriov_totalvfs
echo 8 > /sys/bus/pci/devices/0000:3b:00.0/sriov_numvfs
lspci -D | grep -i 'Virtual Function'
echo 0 > /sys/bus/pci/devices/0000:3b:00.0/sriov_numvfs

The PCI core validates the requested count, programs the SR-IOV capability, enumerates VFs, creates a struct pci_dev for each, and matches each to a VF driver. PF/VF relationships appear in sysfs through virtfn* and physfn. Many platforms require disable-to-zero before changing count.

What happens inside the kernel

PF driver → pci_enable_sriov() ↓ PCI core validates capability and resources ↓ hardware exposes VF functions ↓ struct pci_dev per VF ↓ VF driver binds (or vfio-pci for assignment)

The VF driver normally talks to hardware queues directly, while some configuration requests travel through a VF mailbox to PF driver or firmware. PF reset or firmware failure can therefore disrupt every VF.

VF transmit and receive

On transmit, a guest or host VF driver writes descriptors, rings a doorbell, and the NIC DMA-reads packet data. Hardware applies MAC/VLAN and scheduling policy, then emits the frame. On receive, NIC classification selects a VF queue, DMA-writes into the VF-owned memory, and raises the VF interrupt. The host's normal bridge, iptables, and socket path may never create an skb for that traffic.

wire → NIC parser / hardware classifier → VF RX queue → DMA into guest memory → VF IRQ → guest driver → guest stack

PF policy: identity, VLAN, and trust

ip link set dev eth0 vf 0 mac 02:00:00:00:00:10
ip link set dev eth0 vf 0 vlan 100
ip link set dev eth0 vf 0 spoofchk on
ip link set dev eth0 vf 0 trust off
ip link set dev eth0 vf 0 state auto

Spoof checking prevents a VF from transmitting with an unauthorised source MAC. VLAN policy constrains tenant placement. Trusted VFs may request broader MAC, VLAN, multicast, flow, or promiscuous behaviour and should be treated as infrastructure, not ordinary tenants.

VFIO and IOMMU

For VM assignment, a VF can bind to vfio-pci. VFIO uses the IOMMU to restrict DMA to memory assigned to that VM:

VF 0 → IOMMU domain A → VM A memory VF 1 → IOMMU domain B → VM B memory

modprobe vfio-pci
echo vfio-pci > /sys/bus/pci/devices/0000:3b:01.0/driver_override
echo 0000:3b:01.0 > /sys/bus/pci/devices/0000:3b:01.0/driver/unbind
echo 0000:3b:01.0 > /sys/bus/pci/drivers_probe
lspci -k -s 0000:3b:01.0

SR-IOV partitions device functions; IOMMU supplies DMA isolation. Neither makes VFs independent hardware.

Kubernetes allocation path

Kubernetes does not natively schedule “VF 0000:3b:01.3.” Open-source components divide the job:

Component	Job
SR-IOV device plugin	Discovers VFs and advertises resources such as `intel.com/sriov_netdevice`.
Multus	Attaches an additional network and passes allocated device ID onward.
SR-IOV CNI	Finds the VF netdevice, moves it into the Pod namespace, applies MAC/VLAN/trust policy, IPAM, and cleanup.

scheduler → kubelet → device plugin allocates PCI BDF
          → Multus → SR-IOV CNI → Pod sees eth1

Unlike a veth, a VF can bypass host bridge, kube-proxy, ordinary host firewall hooks, and standard service-mesh observability. That is why SR-IOV is usually a specialised secondary network for telecom functions, storage, packet gateways, DPDK, RDMA, or latency-sensitive appliances.

Production failure patterns

PF reset: many workloads fail together

A firmware hang or PF reset can make every VF lose mailbox communication and link. Kubernetes may still report Node Ready and Pod Running while traffic is dead. Monitor PF health, PCIe AER errors, reset counters, VF link, device-plugin health, and end-to-end traffic.

Stale VF state

If Pod A leaves MAC, VLAN, spoof-check, or trust state behind, Pod B can inherit broken connectivity or unsafe policy. CNI deletion must restore a baseline before returning a VF to the allocatable pool.

Observability gap

Host tcpdump and TC may see nothing. Use PF/VF statistics, ethtool -S, switch telemetry, representors where available, application metrics, PCI error logs, and a traceable map from Pod → VF PCI BDF → netdevice → PF → physical port.

Trade-offs and tests

Performance: fewer host copies, software-switching steps, and scheduler crossings; lower CPU and latency.
Mobility: direct device assignment complicates live migration.
Visibility: host firewalling, capture, policy, and tracing become harder.
Capacity: VFs consume queues, MSI-X vectors, BAR space, firmware objects, packet buffers, and PCIe bandwidth.
Availability: all VFs share one physical failure domain.

Before production, test VF create/disable/reuse, Pod deletion during CNI ADD, node and kubelet restart, device-plugin restart, link flap, PF reset, firmware recovery, VM reboot, spoof/VLAN isolation, throughput fairness, and actual traffic—not only Pod phase.

lspci -Dnnk
cat /sys/bus/pci/devices/0000:3b:00.0/sriov_numvfs
ls -l /sys/bus/pci/devices/0000:3b:00.0/virtfn*
readlink /sys/bus/pci/devices/0000:3b:01.0/physfn
ip -s link show dev eth0
ethtool -S eth0
dmesg | grep -Ei 'sriov|vf|iommu|aer|reset|mailbox'

Mental model: SR-IOV gives one NIC many PCI datapath functions. PF controls. VF carries. NIC switches. IOMMU isolates DMA. Kubernetes assigns ownership. Silicon, firmware, cable, and failure domain remain shared.