Linux event notification / readiness semantics

Linux epoll edge-triggered mode: the ready list, the drain loop, and the event that never comes back

Edge-triggered epoll is fast because it lets the kernel report a readiness change once and leaves state management to user space. That bargain is easy to break: read only part of a socket, wait for another event, and the remaining bytes may sit there forever.

Linux deep dive · Event-driven I/O

The concrete problem

Suppose an event loop watches a non-blocking TCP socket. A packet arrives containing 4 KiB. The loop receives an EPOLLIN notification, calls read(fd, buf, 1024), processes those 1024 bytes, and then goes back to epoll_wait(). The socket still has 3 KiB available, but the loop can sleep indefinitely if no new packet arrives.

This is not a lost packet and not usually a kernel bug. It is the intended contract of EPOLLET: the notification describes a change in readiness, not a continuously maintained report of the current state. If the application does not consume the state change completely, it owns the consequences.

What “readiness” means

epoll is not an asynchronous read operation. It is a wait queue built around the same readiness test used by poll(). A watched file supplies a poll() method; for a TCP socket, that method examines receive queues, shutdown state, errors, and send-buffer space, then returns a bitmask such as POLLIN, POLLOUT, POLLHUP, or POLLERR.

epoll_create1() returns a file descriptor representing an epoll instance. The application adds a watched file with epoll_ctl(EPOLL_CTL_ADD, ...), supplying an interest mask and an opaque data value. Later, epoll_wait() copies ready events into an application-owned array. The returned data is a label chosen by the application; it is not a second lookup performed by the kernel.

How the control and data flow fit together

The important separation is between registration and delivery. Registration attaches epoll’s callback to the watched file’s wait queue. Delivery later scans a private batch of ready items, rechecks the file, and copies event records to user space.

The re-poll during delivery matters. A callback can race with the scan, a read can consume data before delivery, or the source can be closed. epoll does not blindly trust the fact that an item was once queued; it asks the underlying file for its current mask before returning the event.

The open-source implementation

The main implementation is fs/eventpoll.c. Names and locking details change across kernel releases, but the core design is stable.

The current source describes rdllist as a doubly linked FIFO and ovflist as a singly linked LIFO used while the ready list is being scanned without holding the ready-list spinlock. This is an implementation detail with an important purpose: the kernel can copy events to user space and run file poll methods without holding a lock that wakeups may need.

Why the drain loop is a correctness requirement

For a non-blocking stream, the only robust ET read handler looks conceptually like this:

for (;;) {
    ssize_t n = read(fd, buf, sizeof buf);

    if (n > 0) {
        consume_bytes(buf, (size_t)n);
        continue;                 /* look for more data */
    }
    if (n == 0) {
        peer_closed = true;       /* EOF: close or finish the protocol */
        break;
    }
    if (errno == EAGAIN || errno == EWOULDBLOCK)
        break;                    /* drained current readiness */
    if (errno == EINTR)
        continue;                  /* retry without declaring failure */
    report_io_error(errno);
    break;
}

“Drain” means drain the kernel’s current readable state, not necessarily read the entire application message. A protocol parser may stop at one complete frame and leave later frames buffered, but the event loop must either keep reading until EAGAIN or deliberately record that it still owns pending work and process it before sleeping again. A fixed per-turn byte budget can be a valid fairness policy, but then the loop must re-arm its own work queue; it cannot assume ET will remind it.

The same idea applies to writes. Register EPOLLOUT only when an output queue has bytes waiting. On an ordinary TCP socket, “can accept some bytes” is often true, so permanently watching writable readiness can produce a hot loop that starves reads. Under ET, write until EAGAIN, remove EPOLLOUT when the user-space output queue becomes empty, and add it again when a later write cannot complete.

LT, ET, and ONESHOT are different tools

ET does not mean “one event per packet” and does not guarantee one event per transition. Coalescing is allowed. Multiple writes may cause one wakeup, and a wakeup may report several bits. Treat an event as permission to inspect and drain, not as a count of messages.

Concurrency, memory ordering, and descriptor lifetime

The callback can run in a context that cannot sleep, so the ready-list path uses an IRQ-safe spinlock. The delivery path may call copy_to_user() and file poll methods, so it uses the epoll mutex and a task-private scan batch. The overflow list closes the race between those two activities: a wakeup during delivery is recorded and merged after the scan instead of being lost.

An epoll registration refers to an open file, not merely to an integer copied from user space. Closing one descriptor number does not necessarily remove the watched open file if a duplicate descriptor still refers to it. Conversely, reopening a file and receiving the same integer creates a different open-file identity. Safe event-loop designs therefore avoid treating a stale data.fd value as the full identity of a connection.

When events are handed to a worker, use an explicit ownership protocol. A common pattern is a connection object with a generation number or state flag: the event carries a pointer or token, the owner validates that the connection is still live, and close/reuse invalidates the generation before freeing memory. This prevents a delayed event from being interpreted as belonging to a new connection that happened to reuse the same descriptor number.

Failure modes and security implications

Readiness is an input-validation boundary. A peer can send a stream of tiny bytes, keep a connection readable forever, or advertise a protocol frame larger than the service’s limits. The event loop should cap per-connection work, bound parser buffers, enforce idle and total deadlines, and treat malformed framing as a protocol failure. ET improves notification efficiency; it does not provide back-pressure or fairness by itself.

EPOLLWAKEUP has power-management implications: when permitted, it keeps the system awake around event processing. EPOLLEXCLUSIVE can reduce thundering-herd wakeups for multiple epoll waiters, but it has restrictions on which flags combine with it. These are policy features, not substitutes for correct readiness handling.

Debugging method

1. Record every epoll_wait() result with the fd/token, event bits, and the handler’s bytes read or written. A missing callback becomes a timeline problem instead of a hunch.
2. For each ET handler, prove the termination cases: positive progress, EOF, EAGAIN, EINTR, and a real error.
3. Temporarily switch the registration to LT. If the bug disappears, the source likely remains ready after the handler returns; inspect the drain and rearm contract.
4. Use strace -f -tt -e trace=epoll_wait,epoll_ctl,read,recv,write,send,close on Linux to correlate kernel-visible I/O with application state.
5. Use bpftrace, ftrace, or a small counter in the handler to distinguish “no callback” from “callback returned but parser made no progress.”
6. Test short reads and short writes deliberately. Pipe capacity, socket send buffers, signals, and a slow peer expose bugs that a fast loopback client hides.

# Useful Linux observations for a process whose PID is $PID
strace -f -tt -p "$PID" \
  -e trace=epoll_wait,epoll_ctl,read,recvfrom,recvmsg,write,sendto,sendmsg,close

# Inspect the process's descriptors and epoll registrations.
ls -l /proc/"$PID"/fd
cat /proc/"$PID"/fdinfo/<epoll-fd>

# Show the epoll-related syscalls without following child processes.
perf trace -p "$PID" -e epoll_wait,epoll_ctl,read,write

/proc/<pid>/fdinfo/<epoll-fd> can show registered target fds and event masks on Linux, which is useful when the application’s own connection table has drifted from the kernel’s view. The exact fields vary by release.

Compact reproducible lab

Goal

Observe the central ET failure directly: one notification, a partial read, and no second notification until the underlying file is woken again.

Prerequisites

A Linux host or VM with a C compiler and the standard headers. No root access is required. The program uses a non-blocking pipe so the state transition is deterministic and does not depend on TCP timing.

Exact commands

cat > /tmp/epoll-et-lab.c <<'C'
#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <sys/epoll.h>
#include <unistd.h>

static void nonblocking(int fd) {
    int flags = fcntl(fd, F_GETFL, 0);
    if (flags < 0 || fcntl(fd, F_SETFL, flags | O_NONBLOCK) < 0) {
        perror("fcntl"); _exit(1);
    }
}

int main(void) {
    int p[2], ep = epoll_create1(EPOLL_CLOEXEC);
    struct epoll_event add = {.events = EPOLLIN | EPOLLET, .data.fd = 0};
    struct epoll_event out[4];
    char buf[2];
    if (pipe(p) < 0 || ep < 0) { perror("pipe/epoll_create1"); return 1; }
    nonblocking(p[0]);
    if (epoll_ctl(ep, EPOLL_CTL_ADD, p[0], &add) < 0) { perror("epoll_ctl"); return 1; }

    write(p[1], "abcdefghij", 10);                 /* one wakeup, 10 bytes */
    int n = epoll_wait(ep, out, 4, 1000);
    printf("first epoll_wait: %d event(s)\n", n);
    ssize_t r = read(p[0], buf, 1);                 /* intentionally partial */
    printf("partial read: %zd byte(s), first='%c'\n", r, buf[0]);
    n = epoll_wait(ep, out, 4, 200);
    printf("second epoll_wait, no new write: %d event(s)\n", n);

    write(p[1], "Z", 1);                           /* causes another wakeup */
    n = epoll_wait(ep, out, 4, 1000);
    printf("third epoll_wait, after one new byte: %d event(s)\n", n);
    return 0;
}
C
cc -O2 -Wall -Wextra /tmp/epoll-et-lab.c -o /tmp/epoll-et-lab
/tmp/epoll-et-lab

Observations and expected result

On Linux, the output should be equivalent to:

first epoll_wait: 1 event(s)
partial read: 1 byte(s), first='a'
second epoll_wait, no new write: 0 event(s)
third epoll_wait, after one new byte: 1 event(s)

The pipe still contains nine bytes after the partial read, but epoll does not manufacture another ET event merely because the file remains readable. The later write wakes the pipe wait queue again, so the callback can put the epoll item back on the ready list. A correct handler would instead keep reading until EAGAIN.

Revealing variations

Change EPOLLIN | EPOLLET to EPOLLIN. The second epoll_wait() should now return an event because LT requeues an item whose poll mask is still readable. Then replace the one-byte read with a loop until EAGAIN; the loop consumes the pipe and makes the second wait time out even in ET mode.

Environment result: this run is on macOS, where <sys/epoll.h> and Linux epoll are unavailable. The source file was not compiled here; attempting this lab on macOS fails at compilation with a missing epoll header. Use a Linux VM, container, or host for the exact experiment. A macOS fallback is kqueue, but its filter semantics are not identical, so it should not be used to validate Linux ET behavior.

Production example: a worker-based HTTP reactor

A high-concurrency HTTP server may have one reactor thread that owns epoll and a pool of workers that parse requests and produce responses. The reactor registers client sockets as non-blocking ET + ONESHOT. When a worker receives a token, it drains input until EAGAIN, parses complete requests, drains output until EAGAIN, updates the connection’s state, and rearms with EPOLL_CTL_MOD.

The ordering is important. The worker must not rearm while another worker can still mutate the same connection, and it must not free the connection object while a delayed event can still refer to it. A bounded work budget keeps one peer from monopolizing a worker; if more work remains, the connection is placed on a user-space ready queue before rearming or before handing ownership back to the reactor.

When diagnosing such a service, compare three rates: epoll events delivered, successful non-blocking I/O operations, and application progress such as complete HTTP requests. A healthy event rate with zero progress points to parser or ownership bugs; zero events while the socket is readable points to a drain/rearm/lifetime bug; a huge event rate with tiny writes may indicate permanently enabled EPOLLOUT.

Primary sources

• Linux epoll(7) manual
• Linux epoll_ctl(2) manual
• Linux epoll_wait(2) manual
• Linux source: fs/eventpoll.c
• Linux UAPI: epoll flags and event structure
• The design and implementation of the FreeBSD kqueue event notification mechanism (comparison context)

Remember: an ET event is a handoff of responsibility. Once epoll tells you that a source became interesting, keep working that source until it says “not now” (EAGAIN), “finished” (EOF), or “failed” (a real error)—then make an explicit decision about fairness, ownership, and rearming.