Observability / reliability deep dive

Observability / reliability deep dive

2026-09-21

Queue backpressure: why a 10% scheduler bug became hours of publishing delay

A media pipeline can be healthy on average and still fail during a burst. The failure mechanism is queueing: demand briefly exceeds service capacity, background work consumes the little headroom that remains, and a small throughput loss turns into a backlog whose drain time is measured in hours.

1. The concrete problem

When a creator uploads a video episode, the upload is not the publish event. The system must persist the object, enqueue work, transcode it into playback formats, run analysis, update metadata, and expose the finished result to clients. Each stage can be asynchronous because transcoding takes longer than an HTTP request can reasonably wait.

That design creates a user-visible contract: after the upload is accepted, the episode should move through the pipeline within a bounded time. The dangerous case is a queue that accepts work faster than workers can finish it. Nothing is necessarily crashed. Processes are alive, health checks pass, and requests may still return 200. The user experiences “it is missing” because queue age grows silently.

A useful production incident makes this concrete. Spotify described a June 24 publishing delay in which video transcoding reached capacity. A batch job competed with new episodes, a quality change increased per-item processing cost, and a scheduling bug underused available hardware by about 10%. Episodes that normally appeared within minutes waited for hours. The response stopped the batch job, fixed scheduling, added capacity, and later increased transcoding capacity by about 67%.[1]

2. Data flow and the queueing model

Model one stage as a queue with arrival rate λ and service rate μ. If λ < μ over a long enough interval, the queue can drain. If λ > μ, backlog increases at approximately λ − μ items per second. The queue does not care whether the excess came from a launch, a batch job, larger files, or a bad scheduler.

For a single FIFO queue, waiting time follows the intuition captured by queueing theory: as utilization approaches 100%, small changes in arrival rate or service time produce very large changes in delay. The exact distribution depends on burstiness and service-time variance, but the operational rule is robust: running at 99% average utilization leaves almost no room for a burst or a slow item.

For a pipeline, the slowest constrained stage sets the useful throughput. Adding database replicas does not help if video workers are full. Adding workers does not help if object storage throttles. The first debugging question is therefore not “which service is red?” but “which stage has the oldest work, and what resource gates its service rate?”

3. How small capacity losses cascade

Suppose normal arrivals are 100 jobs per minute and the transcoder can process 110. Ten jobs per minute of headroom sounds comfortable. Now combine four ordinary changes:

1. A burst raises arrivals to 125 jobs per minute.
2. A batch job consumes 15 worker slots.
3. A codec improvement raises average CPU time by 10%.
4. A scheduler bug leaves 10% of the remaining slots idle.

The arithmetic is deliberately simple, but the shape matters: each change attacks a different side of the capacity equation. The burst raises λ; the codec change lowers per-worker μ; the batch job removes capacity; the scheduler bug prevents the remaining capacity from being used. Their effects multiply rather than politely taking turns.

Depth alone is a weak alert. A queue with 10,000 tiny jobs may be healthier than one with 100 large jobs. Alert on age, oldest-item age, arrival-versus-completion rate, and estimated drain time. A simple estimate is drain_time ≈ backlog / (service_rate − arrival_rate), valid only while the rates remain stable. When arrival rate is at or above service rate, the estimate correctly tells you there is no drain.

4. Priority, backpressure, and headroom

Priority is a resource policy, not a label

New publication work and reprocessing old episodes may use the same transcoding pool, but they do not have the same deadline. A fair scheduler that gives each queue equal service can still violate the product promise for fresh uploads. Use explicit classes, such as “publication” and “maintenance,” then reserve capacity or apply weighted fair queuing so maintenance cannot starve publication.

Kubernetes Jobs make the same distinction at a different layer: a Job represents work that should finish, while a work-queue Job lets parallel workers fetch tasks from an external queue. The controller tracks completion, but it does not magically choose the business-priority policy inside that queue.[3] That policy belongs in the worker system.

Backpressure has to reach the producer

If a producer can enqueue unlimited work, a downstream queue becomes an unbounded memory and storage bill. Backpressure means the producer sees a bounded signal: reject, delay, shed, or downgrade work when the consumer cannot keep up. For uploads, that can mean accepting the bytes durably while returning an explicit “processing delayed” state, rather than pretending publication is imminent.

Do not implement backpressure with blind retries. A client retrying a missing episode can create a second job while the first is still in the queue. Use an idempotency key derived from the creator’s upload identity, return the existing job state, and make retry timing exponential with a cap and jitter.

Headroom must include recovery

Capacity planning based only on steady-state traffic misses the worst moment: recovery. When workers return after an outage, the backlog is largest and every queued item competes for service. Keep enough spare capacity to drain backlog while new traffic continues. Spotify’s follow-up engineering report explicitly described adding burst capacity, workload prioritization, rate limiting, and backpressure as part of the broader reliability work.[2]

5. Debugging the incident

Start with a time-aligned graph, not a list of red dashboards. Plot arrivals, completions, queue depth, oldest age, worker concurrency, CPU utilization, batch occupancy, retries, and deploy/configuration changes on one UTC timeline.

1. Find the first divergence. The first sustained gap between arrivals and completions is usually more useful than the first user report.
2. Partition by queue and priority. A healthy low-priority queue can hide a starving publication queue.
3. Check effective capacity. Compare requested worker concurrency with actual running workers, CPU throttling, resource limits, and scheduler decisions.
4. Measure service time. If the p95 transcode time grew after a codec change, the old capacity model is obsolete.
5. Inspect retries and duplicates. Recovery traffic can keep the queue growing after the original fault is fixed.
6. Test drain math. Estimate whether the proposed fix makes completion rate exceed arrival rate by enough to clear the backlog within the user contract.

The Spotify timeline is a useful lesson in detection quality: early alerts fired at 13:30, but the broader capacity incident was not recognized until queue thresholds breached at 17:34. An alert that says “capacity is high” is weaker than one that says “publication queue age will exceed 15 minutes in 12 minutes at current rates.”

6. Reproducible lab

Goal: watch a small service-time regression turn a stable queue into a backlog, then recover it with priority and admission control.

Prerequisites: Python 3.11 or newer. No third-party packages.

mkdir -p /tmp/queue-lab
cd /tmp/queue-lab
cat > queue.py <<'PY'
from collections import deque
import sys

arrival = int(sys.argv[1]) if len(sys.argv) > 1 else 100
service = int(sys.argv[2]) if len(sys.argv) > 2 else 110
minutes = int(sys.argv[3]) if len(sys.argv) > 3 else 30
batch = int(sys.argv[4]) if len(sys.argv) > 4 else 0

publish = deque()
maintenance = deque()
for minute in range(minutes):
    incoming = arrival * (3 if 8 <= minute < 12 else 1)
    for _ in range(incoming):
        publish.append(minute)
    capacity = max(0, service - batch)
    done = 0
    # Publication work gets the first slice of every minute.
    while publish and done < capacity:
        publish.popleft(); done += 1
    while maintenance and done < capacity:
        maintenance.popleft(); done += 1
    if minute % 5 == 0 or minute == minutes - 1:
        age = minute - publish[0] if publish else 0
        print(f"t={minute:02d} publish={len(publish):5d} oldest_age={age:2d}m done={done:3d}")
PY
python3 queue.py 100 110 30 0
python3 queue.py 100 110 30 15
python3 queue.py 100 99 30 0

Expected result: the first run absorbs the short burst because it has 10 jobs per minute of headroom. The second run shows that a batch consumer can remove the headroom and leave publication work waiting. The third run never reaches a sustainable drain rate because service capacity is below normal arrivals; queue age keeps increasing.

Variation 1: change the worker policy so maintenance runs before publication. The total work may be unchanged, but publication age becomes much worse. This demonstrates why priority is a user-facing contract.

Variation 2: add an admission limit, for example stop appending when len(publish) > 500. Report “processing delayed” instead of accepting unlimited work. The queue stops growing, but the producer now needs an explicit status and retry contract.

Lab limitation: this model uses integer jobs and a deterministic minute clock. It does not model variable transcode time, worker crashes, or storage throttling. Use it to reason about rate and priority, not to predict a real fleet’s exact SLO.

7. Production design and security

Make queue state durable and idempotent. A worker crash must not lose a job, and a retry must not publish the same episode twice. Keep a job identity, attempt number, lease expiry, and terminal state. A queue item should move through explicit states such as accepted → processing → published or failed → retryable/dead-letter.

Protect the control surface. Queue priority, worker concurrency, codec settings, and admission limits are operational policy. An attacker who can submit huge files, force expensive codecs, or change priority can consume capacity without exploiting a memory-safety bug. Rate-limit by account and tenant, cap resource-intensive dimensions, authenticate control-plane changes, and audit who changed worker allocation.

Finally, make the contract observable end to end. Record an upload ID across ingest, queue, worker, storage, and publish events. Expose queue age by priority and tenant, not just aggregate depth. Sample completed and failed traces. A creator should not be the first monitoring system to discover that a queue has been stuck for four hours.

Takeaway

A queue is not a buffer that makes capacity problems disappear. It converts overload into time. If arrivals briefly exceed service, headroom determines whether the queue drains quietly or becomes a multi-hour incident. The practical discipline is to measure age, preserve priority, apply backpressure at admission, reserve capacity for recovery, and verify that every “fix” makes completion rate exceed arrival rate. The 10% scheduler bug mattered because the system had already spent its margin elsewhere.