Linux IPC / pipes and concurrent writers

Linux pipe atomicity: why PIPE_BUF protects records but pipe capacity does not

A pipe is a byte stream, not a queue of messages. Linux nevertheless gives small writes an atomicity guarantee, which is just enough to build simple record protocols—provided you do not confuse that guarantee with the pipe’s total buffering capacity.

Linux deep dive · IPC and record framing

The concrete problem

Imagine two worker processes writing log records to one pipe that a collector reads. Each worker calls write(fd, record, length). If the records are short, the collector should not receive half of worker A’s record followed by half of worker B’s. If the records are large, however, the pipe may fill while a write is in progress. At that point, blocking, partial progress, and interleaving become possible.

Linux exposes two different sizes that are easy to mix up:

What the POSIX contract actually says

POSIX defines PIPE_BUF as the maximum number of bytes that can be written atomically to a pipe. “Atomic” here means that the bytes from one write are not interleaved with bytes from another writer. It does not mean the reader’s read() call returns exactly one record, and it does not mean a record is preserved as a unit for later reads.

The four cases are determined by write size and whether the file descriptor is non-blocking:

The “all bytes fit” detail is intentionally stronger than “the kernel will eventually copy a small write.” A blocking small write waits rather than returning a short record. With non-blocking I/O, the application must handle EAGAIN and short writes itself.

A reader has no corresponding record guarantee. It may read fewer bytes than one atomic write, or several atomic writes in one call. A protocol that needs records must include framing—newline-delimited records, a length prefix, or a fixed-size structure—and must handle a record split across reads.

Linux’s implementation model

In Linux, an anonymous pipe is backed by a special inode and a ring of struct pipe_buffer entries. Each entry refers to a segment of memory containing pipe data. The ring has a logical head and tail; writers reserve space at the head, readers consume from the tail.

The ring is a storage mechanism, not a message queue. One logical write can occupy multiple pipe_buffer entries, and one read can consume only part of an entry. Atomicity is enforced by the write-side locking and accounting rules for small writes; it is not supplied by the existence of a one-buffer-per-message boundary.

Current kernel source has evolved through page and folio changes, so exact helper names can move. The stable path is in fs/pipe.c: pipe_write() handles user writes, pipe_read() handles reads, and pipe_wait() is part of the blocking/wakeup path. The pipe object itself is described by struct pipe_inode_info in include/linux/pipe_fs_i.h.

The control flow of a blocking write

1. The writer enters pipe_write() and checks whether any reader file descriptions remain. If not, the kernel sends SIGPIPE and the call returns EPIPE if the signal is ignored or handled.
2. The kernel takes the pipe’s lock and examines free ring space and the write size.
3. If a small write cannot fit, the writer drops the lock and sleeps on the pipe wait queue. It must re-check the condition after waking; a wakeup is only a hint because another writer may have won the space.
4. Once the write can progress, Linux copies data from the writer’s address space into pipe-backed storage. The writer may be interrupted or encounter a fault while copying from user memory.
5. The kernel publishes the newly available bytes, releases the lock, and wakes readers and possibly other writers.

For a write larger than PIPE_BUF, the kernel is allowed to make partial progress before sleeping and resuming. That is where another writer can get a turn. For a write at or below PIPE_BUF, Linux preserves the all-or-wait behavior for blocking descriptors, so the call does not publish a partial record merely because the pipe is temporarily full.

This is also why a naive “one call to write() equals one message” assumption fails in both directions: small writes are not message-framed for readers, while large writes are not protected from interleaving among writers.

Important structures, constants, and functions

Do not use F_GETPIPE_SZ as a substitute for PIPE_BUF. On a usual Linux system, capacity is much larger than the atomicity limit. The distinction is deliberate: buffering controls throughput and backpressure; atomicity controls interference between concurrent writers.

Tradeoffs and failure modes

Pipe capacity is also a resource-accounting issue. Linux applies per-user pipe-page limits and system-wide memory pressure can make capacity growth fail. An application that creates many pipes and enlarges all of them can consume substantial unswappable kernel memory. Treat capacity as a bounded queue, not as an infinite extension of userspace RAM.

Security and isolation implications

Pipe atomicity is not authentication, authorization, or integrity. If an untrusted process has a write descriptor, it can inject syntactically valid records, fill the pipe, or close descriptors in ways that change the reader’s failure behavior. Validate record lengths and contents after framing; never let a length prefix cause an unbounded allocation.

Descriptor inheritance is a common security and availability bug. After fork(), every process that retains a duplicate write end keeps the reader from observing end-of-file. Set O_CLOEXEC at creation where possible, close unused ends immediately after fork(), and audit subprocess launch paths. A leaked descriptor can also keep sensitive data reachable longer than intended.

A producer that can force a privileged consumer to process unbounded records can create CPU and memory denial of service. Bound record size, bound the number of queued records, apply timeouts or cancellation, and decide explicitly whether the writer should block, drop, or shed work when the pipe is full.

Debugging method

1. First determine whether the symptom is interleaving, partial progress, or backpressure. They are different bugs and need different evidence.
2. Trace system calls with strace -ff -e trace=read,write,fcntl,close. Look for short writes, repeated EAGAIN, long blocked calls, SIGPIPE, and a missing close on an inherited descriptor.
3. Inspect capacity with fcntl(fd, F_GETPIPE_SZ). For a process-level view on Linux, /proc/<pid>/fdinfo/<fd> can identify the descriptor, but it does not expose a message count.
4. Instrument the protocol with sequence numbers and writer IDs. Do not infer corruption from a single read() returning two records; that is normal stream behavior.
5. Reproduce under pressure: use several writers, small pipe capacity, a deliberately slow reader, and non-blocking mode. Scheduler timing often hides the bug in a quiet test.
6. When inspecting kernel behavior, read the version-matched fs/pipe.c. Generic web searches frequently show stale helper names or explain POSIX semantics without describing Linux’s current ring implementation.

# Linux tracing examples
strace -ff -tt -e trace=read,write,fcntl,close -o /tmp/pipe.trace ./pipe-lab
rg -n 'write\(|read\(|EAGAIN|EPIPE|SIGPIPE|F_SETPIPE_SZ' /tmp/pipe.trace*

# Source-oriented search in a checked-out Linux tree
rg -n 'struct pipe_inode_info|struct pipe_buffer|pipe_write|pipe_read|pipe_wait' fs/pipe.c include/linux/pipe_fs_i.h

Compact reproducible lab

Goal

Run two concurrent writers against one pipe. In the small-record mode, each record is written atomically but the reader still receives an ordinary byte stream. In the large-record mode, the program makes the pipe small and slows the reader so that a single large write must yield, exposing interleaving.

Prerequisites

Linux with a C compiler is the reference environment. The source also compiles on macOS, but macOS does not provide Linux’s F_SETPIPE_SZ; the program will use the platform’s default capacity, so the large-record variation may be less deterministic there.

Exact commands

cc -O2 -Wall -Wextra -o /tmp/pipe-atomicity -x c - <<'C'
#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/wait.h>
#include <unistd.h>

static void die(const char *what) { perror(what); exit(1); }

static void writer(int fd, char mark, size_t n, int rounds) {
    char *buf = malloc(n);
    if (!buf) die("malloc");
    memset(buf, mark, n);
    for (int r = 0; r < rounds; r++) {
        size_t sent = 0;
        while (sent < n) {
            ssize_t w = write(fd, buf + sent, n - sent);
            if (w < 0) {
                if (errno == EINTR) continue;
                die("write");
            }
            sent += (size_t)w;
        }
    }
    free(buf);
    close(fd);
    _exit(0);
}

static void consume(int fd, size_t write_size, size_t expected_total, int slow) {
    char buf[4096];
    char previous = 0;
    size_t run = 0, total = 0, short_runs = 0;
    for (;;) {
        ssize_t r = read(fd, buf, sizeof buf);
        if (r == 0) break;
        if (r < 0) {
            if (errno == EINTR) continue;
            die("read");
        }
        for (ssize_t i = 0; i < r; i++) {
            char c = buf[i];
            if (c != 'A' && c != 'B') die("unexpected byte");
            if (c != previous) {
                if (previous != 0 && run < write_size) short_runs++;
                previous = c;
                run = 1;
            } else {
                run++;
            }
            total++;
        }
        if (slow) usleep(1000);
    }
    if (run < write_size) short_runs++;
    printf("received=%zu expected=%zu runs-shorter-than-write=%zu\n",
           total, expected_total, short_runs);
}

int main(int argc, char **argv) {
    int large = argc > 1 && !strcmp(argv[1], "large");
    size_t n = large ? 65536 : 64;
    int rounds = large ? 8 : 1000;
    int p[2];
    signal(SIGPIPE, SIG_DFL);
    if (pipe(p) < 0) die("pipe");
#ifdef F_SETPIPE_SZ
    if (large && fcntl(p[1], F_SETPIPE_SZ, 4096) < 0)
        perror("F_SETPIPE_SZ (continuing with default capacity)");
#endif
    long atomic_limit = fpathconf(p[1], _PC_PIPE_BUF);
    if (atomic_limit < 0) die("fpathconf");
    printf("atomic-limit=%ld write-size=%zu\n", atomic_limit, n);
    fflush(stdout);

    pid_t a = fork();
    if (a < 0) die("fork");
    if (a == 0) { close(p[0]); writer(p[1], 'A', n, rounds); }
    pid_t b = fork();
    if (b < 0) die("fork");
    if (b == 0) { close(p[0]); writer(p[1], 'B', n, rounds); }
    close(p[1]);
    consume(p[0], n, n * (size_t)rounds * 2, large);
    close(p[0]);
    waitpid(a, NULL, 0); waitpid(b, NULL, 0);
    return 0;
}
C

# Small writes: each 64-byte write is below PIPE_BUF.
/tmp/pipe-atomicity

# Large writes: each 65536-byte write exceeds PIPE_BUF and the pipe is pressured.
/tmp/pipe-atomicity large

# Optional Linux syscall trace for one run:
strace -ff -tt -e trace=read,write,fcntl,close /tmp/pipe-atomicity large

The program counts runs of identical bytes shorter than the individual write size. It is not pretending that a run is a complete application record; it is a deliberately simple probe of whether the two writers’ byte ranges were interleaved.

Expected result

On Linux, the small run should report runs-shorter-than-write=0 or a similarly low value caused only by the final stream boundaries; every 64-byte write is protected from another writer. The large run should report a positive count, often many short runs, because 65536 bytes cannot be committed as one atomic write and the slow reader creates opportunities for the writers to alternate.

The total byte count should equal n × rounds × 2 in both modes. That is the important control: interleaving changes order, not necessarily the amount of data.

Revealing variations

Change the small-mode n from 64 to atomic_limit + 1 after printing the queried value. The result crosses the portability boundary: POSIX no longer promises atomicity, even if one particular run happens not to interleave. Then remove the usleep(1000) call; the large mode may show fewer interleavings because the reader drains the pipe quickly, demonstrating that a race can be real without being reliably visible under light load.

Environment result: this host is macOS. The compiler can build the program, but macOS does not implement Linux’s F_SETPIPE_SZ interface, so the forced-small-capacity part of the large-write experiment cannot be reproduced exactly here. Run the commands on a Linux VM or container for the reference result; on macOS, the small-write guarantee is still useful, while the large-mode interleaving result is timing-dependent.

Production example: multi-process logging

A common design is for several preforked workers to write newline-delimited diagnostics to one pipe connected to a log collector. If each line is bounded below PIPE_BUF, writers do not need a shared mutex to keep lines from interleaving. The collector still needs to handle one line split across reads and several lines returned together.

The design starts to fail when stack traces, serialized requests, or batched JSON exceed the atomic limit. A newline inside a payload may then be mistaken for a record boundary, and a pipe-full stall can couple an unhealthy collector to every producer. Practical designs cap record size, escape or length-frame payloads, use one dedicated logging writer, or move to a message-oriented transport with explicit queue limits and drop policy.

The same rule appears in shell pipelines and supervisor protocols. A command that writes a short status line is usually safe against concurrent writers; a command that streams a large binary object must be treated as a byte-stream producer with explicit partial-write handling.

Primary sources

• Linux pipe(7) manual
• Linux pipe(2) manual
• POSIX write(): pipe atomicity requirements
• POSIX PIPE_BUF definition
• Linux source: fs/pipe.c
• Linux source: pipe structures and operations
• Kernel documentation: pipe-user-pages limits
• Linux fcntl(2): F_GETPIPE_SZ/F_SETPIPE_SZ

Remember: PIPE_BUF is an interference boundary between writers. Pipe capacity is a buffering boundary between producers and consumers. One protects record construction; the other applies backpressure.