Cloud platforms / virtualization deep dive

Firecracker’s microVM fast path: where KVM stops and virtio I/O begins

A microVM is not “a container with a nicer process.” It is a guest kernel running through a hardware virtualization boundary, with a deliberately tiny device model on the host side. The important engineering question is where execution crosses that boundary, how a guest request becomes host I/O, and how the same crossing is kept small enough to defend.

Cloud platforms / virtualization deep dive · KVM exits, virtio queues, event-driven device emulation, rate limiting, and layered isolation

2026-08-26

The concrete problem

A public cloud has to run code that is not trusted by the host, often from many customers, while starting and stopping that code quickly. Containers are cheap because processes share a kernel; that shared kernel is also a large cross-tenant attack surface. Traditional virtual machines provide a stronger hardware boundary, but a general-purpose VMM such as QEMU models a large collection of legacy hardware and is correspondingly heavier.

Firecracker is the deliberately narrow middle: one process owns one microVM, KVM executes the guest CPU, and a small Rust VMM exposes only the devices needed by serverless and container workloads. AWS’s August 25, 2026 note marking EC2’s twentieth birthday is a useful reminder of the trajectory: cloud compute keeps moving more of the virtualization boundary into specialized, minimal infrastructure. The lesson here is the mechanism, not a product-history recap.

The boundary in one picture

Consider a guest writing a block. The guest kernel does not issue a host write(2). It places a descriptor chain in guest memory, publishes the chain through a virtio queue, and notifies the virtual device. Firecracker eventually reads those guest descriptors and performs a host operation against a configured backing file or backend.

There are two related but distinct wakeup paths. A synchronous device notification can cause KVM to return a VcpuExit such as an MMIO write, which the VMM handles and then re-enters KVM. A virtio queue is commonly represented by an event file descriptor registered with the VMM’s event manager; when the guest kicks it, the device subscriber drains the queue outside the vCPU’s tight execution loop. Keeping these paths separate prevents device work from becoming accidental guest execution.

What KVM actually contributes

KVM is a Linux kernel interface, not a complete virtual machine. User space opens /dev/kvm, creates a VM and vCPU file descriptor, maps guest memory, configures registers and virtual interrupt state, then repeatedly calls KVM_RUN. Most ordinary guest instructions run directly on the host CPU with hardware virtualization. KVM returns to user space when the guest performs an operation that the VMM must emulate or when an external event interrupts the run.

A VM exit is therefore a control-flow boundary, not a packet or I/O completion by itself. A guest MMIO write may mean “the queue has new work”; the VMM still has to find the queue, read guest memory, validate the request, and arrange completion. Confusing an exit with completed I/O is a common source of latency and correctness bugs.

Virtio queues: shared memory plus an explicit ownership protocol

Virtio avoids emulating a physical disk controller instruction by instruction. The driver and device share a queue in guest memory. A descriptor contains an address, length, and flags; a chain can contain read-only request headers, writable data buffers, and a writable status byte. The driver publishes available descriptor heads, and the device publishes used entries after it has consumed them.

The rings are a protocol for ownership. A descriptor that the guest has made available may be read by the device; a used entry tells the guest it can reclaim the descriptor chain. Memory ordering and interrupt suppression are part of the virtio specification. The device must not trust a guest-provided length merely because it fits in an integer: it must check address arithmetic, descriptor flags, chain depth, alignment assumptions, and whether a supposedly writable buffer is actually writable.

For a block write, the VMM’s high-level flow is:

1. consume a queue notification and pop an available descriptor head;
2. walk the chain from guest memory, rejecting malformed or looping chains;
3. decode the request type and sector, then map the data buffers into host I/O vectors;
4. charge operation and byte tokens before starting the backend request;
5. complete synchronously or wait for an async completion event;
6. write the status and used-ring entry, then trigger the guest interrupt.

Firecracker’s control and data flow

Firecracker intentionally keeps its control plane out of the hot path. The API thread receives configuration such as vCPUs, memory, a block image, and a network interface. It sends a typed VmmAction into the VMM. After the instance starts, the API thread does not execute guest I/O; vCPU threads run KVM and device subscribers react to event sources.

The important design choice is not merely “Rust instead of C.” It is that the VMM has a small, explicit machine model. A guest can execute many instructions without involving Firecracker, but device boundaries are represented by a small set of queue, memory, interrupt, and backend operations. Fewer emulated devices mean fewer state machines and fewer parser-like surfaces reachable from a hostile guest.

Important open-source structs and functions

The following names are from Firecracker commit 761f88f. They are implementation landmarks, not a promise of API stability.

Why the event loop is part of the performance design

A naive VMM could let the vCPU thread process a queue, issue a blocking disk call, and then resume the guest. That is simple but creates bad coupling: a slow disk can stop guest CPU progress, and one busy guest can make device work unfair to other guests. Firecracker instead uses event sources for queue kicks, rate-limiter timers, and async completions. The block subscriber can submit work, return to the event loop, and later complete the same descriptor when the backend signals.

The event loop does not make I/O free. It adds bookkeeping, completion state, and synchronization. Its payoff is that each microVM gets explicit scheduling and resource-control points. A production operator can observe queue depth, completion latency, throttling, and backend errors instead of seeing only an opaque “VM is slow” symptom.

Rate limiting is both fairness and containment

Firecracker’s token bucket has a capacity, a refill time, and an optional one-time burst. Devices can charge separate budgets for bandwidth and operations. A request that arrives when the budget is empty is not necessarily rejected forever: the rate limiter arms a timer and the device is revisited when tokens are available.

That control matters even when the host storage is fast. Without it, a guest can use a valid virtio request stream to monopolize host I/O queues, starve neighboring microVMs, or create a queue explosion in a backend. The limiter also has to handle the awkward case where a single request is larger than the bucket capacity; silently allowing it would make the configured bound meaningless, while splitting it changes completion semantics. Firecracker exposes this as an explicit over-consumption path for the device to handle.

Security boundaries and failure modes

Hardware virtualization is one layer, not the entire security argument. Firecracker’s design adds process-level barriers: the jailer sets up a restricted filesystem and identity, cgroups constrain resources, namespaces reduce visibility, and seccomp filters reduce the system calls available to the VMM threads. The VMM’s own device model must still be correct because a guest can intentionally exercise every reachable device path.

• Guest-memory confusion. A malformed descriptor can cause out-of-bounds reads, a writable/read-only mismatch, or integer overflow in address-plus-length arithmetic. Validate before creating host slices or I/O vectors.
• Completion races. Async I/O can complete after a device is paused, snapshotted, or hot-unplugged. State ownership and cancellation must make it impossible to write a used entry into an invalid queue.
• Interrupt storms. A guest can repeatedly kick a queue or request tiny operations. Coalescing, eventfd draining, rate limits, and queue limits prevent the notification mechanism from becoming the workload.
• Backend escape. A vhost-user or custom backend is another trust boundary. Firecracker’s built-in rate limiting does not automatically protect an external backend; that backend needs its own sandbox and limits.
• Metadata exposure. A host-configured metadata service is convenient for bootstrapping but becomes credential-bearing state. Scope its contents and treat guest access as an intentional capability.
• Kernel dependency. A VMM bug can be amplified by a KVM or host-kernel vulnerability. Keep the host kernel patched, minimize device models, and do not treat a successful seccomp installation as proof of isolation.

Tradeoffs

How to debug a slow or stuck microVM

1. Classify the delay first: guest CPU, KVM exit frequency, queue admission, backend latency, or completion/interrupt delivery. “I/O latency” is too coarse.
2. Correlate Firecracker metrics with host metrics. Look at queue depth, rate-limiter throttles, async completion time, host disk latency, CPU steal, and vCPU exits.
3. Trace the request lifecycle with a descriptor or request identifier that is safe to log. Record queue notification, descriptor validation result, admission decision, backend submission, completion, used-ring publication, and interrupt.
4. When a guest appears hung, inspect whether the used index advances. If it does not, the fault is before completion; if it advances but the guest does not progress, inspect interrupt delivery and guest driver state.
5. For a suspected exit storm, sample KVM_EXIT_MMIO addresses and the guest instruction path. A guest repeatedly touching an unsupported or misconfigured device can look like a storage outage while actually burning vCPU time in the VMM.
6. Reproduce with the smallest device set and a fixed image. Then vary only one dimension—queue size, rate limit, sync versus async backend, or vCPU count—so the trace still explains causality.

Compact reproducible lab: inspect the queue and exit paths

Goal: connect the abstract path to real Firecracker code and run the unit tests that exercise KVM-exit translation, virtio event dispatch, and token-bucket behavior.

Prerequisites: Linux, Git, Rust/Cargo, and a readable /dev/kvm. The tests do not require a cloud account, but a full boot test additionally needs a compatible kernel and root privileges for networking setup.

LAB=$(mktemp -d /tmp/firecracker-lesson.XXXXXX)
git clone --depth 1 https://github.com/firecracker-microvm/firecracker.git "$LAB/firecracker"
cd "$LAB/firecracker"

# Record the source version and confirm the hardware boundary exists.
git rev-parse HEAD
test -r /dev/kvm && echo "KVM available" || echo "KVM unavailable"

# Source-level VMM tests: KVM-exit translation and virtio event dispatch.
cargo test -p vmm --lib vstate::vcpu::tests::test_handle_kvm_exit -- --nocapture
cargo test -p vmm --lib devices::virtio::block::virtio::event_handler::tests::test_event_handler -- --nocapture

# Token accounting does not need to boot a guest.
cargo test -p vmm --lib rate_limiter::tests::test_token_bucket_reduce -- --nocapture

Expected observation on Linux: the tests compile the vmm crate and pass. The first test shows that an MMIO read/write is converted into bus/device work; the second drives a virtio block queue through the event manager and verifies the used entry and status byte; the third checks that the bucket admits, refuses, and replenishes work according to its budget.

What happened in this environment: this host is Darwin/arm64, Cargo is not installed, and /dev/kvm is absent. The commands therefore cannot run here; the first attempted test stopped immediately with zsh: command not found: cargo. That is an environment limitation, not a passing result.

Fallback without Linux: the checkout is still useful for a deterministic source trace:

git grep -n "fn handle_kvm_exit\|KVM_RUN" -- src/vmm/src/vstate/vcpu.rs
git grep -n "PROCESS_QUEUE\|process_queue_event\|PROCESS_ASYNC_COMPLETION" -- src/vmm/src/devices/virtio/block/virtio/event_handler.rs
git grep -n "pub struct TokenBucket\|pub fn reduce\|OverConsumption" -- src/vmm/src/rate_limiter/mod.rs

Revealing variations:

• On Linux, run the virtio event-handler test alone with --exact and add --nocapture; then read the test beside process. Notice that queue notification, activation, and async completion are separate event sources.
• Change the token-bucket test filter to rate_limiter::tests::test_rate_limiter_overconsumption. A single oversized operation is deliberately a different state than ordinary temporary depletion.

The design in one sentence

Firecracker gets microVM density by allowing KVM to execute ordinary guest instructions directly, then making every device crossing a small, observable, rate-limited, and sandboxed user-space state machine. The fast path is not “no virtualization”; it is carefully choosing which operations pay the virtualization boundary and making the paid crossings easy to reason about.

Primary sources

• AWS News Blog: Happy 20th Birthday, Amazon EC2 — the current event that anchors the cloud-platform context.
• Firecracker project overview — supported devices, KVM relationship, startup and density goals, and the microVM rationale.
• Firecracker design document — thread model, virtio devices, rate limiting, jailer, cgroups, namespaces, and seccomp.
• Linux KVM API documentation — the user-space interface for VM creation, vCPU execution, and exits.
• OASIS Virtio 1.2 specification — descriptor tables, available/used rings, device notification, and memory-ordering rules.
• Firecracker vstate source — vCPU and VM state surrounding the KVM boundary.
• Firecracker virtio source — queue, device, block, network, and event-manager implementations.
• Firecracker jailer documentation — the process-level containment layer around the VMM.