Observability and reliability / current incident

GitHub's 7h47m outage: when retries multiply a bottleneck

On August 17, 2026, GitHub reported a 7-hour-47-minute incident in which a service-mesh concurrency limit, an autoscaling policy that watched the wrong signal, and optimistic retries formed a feedback loop. The interesting lesson is not that one component failed. It is how a system can have spare application CPU while its gateway is saturated—and how every well-intentioned retry can make recovery slower.

Reliability deep dive · overload control, service-mesh metrics, and retry storms

2026-08-20

The concrete problem

Imagine an HTTP request passing through a proxy sidecar before it reaches the application:

The application can be healthy while the sidecar is full. New requests then wait, time out, or receive an error before the application does meaningful work. If clients retry those errors, the next wave arrives at the already-full sidecar. The system is now spending capacity on attempts instead of successful requests.

That creates a dangerous mismatch between two kinds of load:

What happened at GitHub

GitHub's official status report says the incident ran from 13:28 to 21:15 UTC on August 17, 2026. At peak, web/API error rates were about 20%, while archive and raw-content downloads reached about 50%. Issues, pull requests, APIs, Actions, authentication flows, and Copilot were affected.

The published causal chain is unusually useful because it names both the resource limit and the control-loop mistake:

Most services recovered when the Central US datacenter recovered at about 16:36 UTC. Actions remained degraded until about 18:03. The Copilot Token Service took until about 21:02 because delayed replies to one internal endpoint triggered a latent retry bug in Visual Studio Code. GitHub reported Copilot token traffic rising from a normal 7–9 thousand requests per second to 70–100 thousand—roughly an order of magnitude. The status report also mentions scraping attacks on codeload endpoints as a complicating factor.

This is a current incident, so details can change as postmortem work continues. The durable mechanism is already clear: a capacity limit was outside the autoscaler’s observation boundary, and retries converted latency into additional offered load.

The feedback loop, quantitatively

Let λ be the rate of new logical requests, p the probability that one attempt succeeds before its timeout, and r the maximum number of retries. If every failed attempt is retried, the expected number of attempts per logical request is:

A = 1 + (1 − p) + (1 − p)² + … + (1 − p)ʳ
  = (1 − (1 − p)ʳ⁺¹) / p

At p = 0.4 and three retries, A = 1 + 0.6 + 0.36 + 0.216 = 2.176. A service receiving 1,000 new requests per second is asked to process about 2,176 attempts per second on average. As p falls under overload, A rises, which adds load and pushes p down again.

Real systems are messier: backoff, jitter, deadlines, connection pooling, and retry budgets change the shape. A client bug can be worse than the simple formula because it may retry a delayed response while the original request is still executing. That is duplicated work, not merely another chance after a failure.

Why the autoscaler missed the bottleneck

Kubernetes Horizontal Pod Autoscaling (HPA) periodically reads a metric and computes a desired replica count. For a utilization target, the central idea is approximately:

desiredReplicas = ceil(currentReplicas × observedUtilization / targetUtilization)

The formula is not the hard part. The hard part is defining observed utilization. In a service-mesh deployment, one Kubernetes Pod can contain both an application container and an Envoy sidecar. The application may report 35% CPU while the sidecar is rejecting work at its concurrency limit. An HPA using only the application container’s CPU has no mathematical reason to add replicas.

Even a sidecar-aware metric can be too slow or too indirect:

• A CPU metric is a poor proxy for a hard queue or connection cap. A proxy waiting on sockets may consume little CPU.
• A five-minute average can hide a burst whose request deadlines are measured in seconds.
• Scaling a backend does not immediately create gateway capacity. New pods need scheduling, image pulls, sidecar warm-up, endpoint publication, and load-balancer convergence.
• Scaling can increase total connections and fan-out. If the next bottleneck is a shared load balancer, database, or egress quota, adding pods can make it worse.
• Regional failure can make a healthy-region autoscaler react to shifted traffic while the failing region continues retrying. Capacity and traffic placement must be coordinated.

For this class of incident, the useful signal is not one magic metric. It is a small set that describes the admission boundary and the work boundary together:

Open-source implementation landmarks

The public incident does not expose GitHub’s complete configuration. The following open-source components show where the same mechanics live in a typical Kubernetes/Istio stack.

1. Kubernetes HPA is a delayed metric-control loop

The Kubernetes controller in pkg/controller/podautoscaler/horizontal.go reads metrics, computes a desired replica count, applies stabilization and tolerance rules, and writes the scale subresource. The metric may come from the resource metrics API, a custom metric, or an external metric. The controller therefore scales what the metric provider exposes—not what the operator mentally considers “the service.”

Important design questions are: which container contributes the metric, how fresh the sample is, what happens when the metric is missing, and whether scale-up is fast enough for the overload’s time constant. A correct HPA formula with a stale or downstream metric is still the wrong controller.

2. Envoy makes queue and retry policy explicit

Envoy sidecars and gateways expose counters and gauges for downstream connections, upstream active connections, pending requests, request timeouts, overflow, and retries. Names vary by listener and cluster, but the useful families include downstream_cx_active, upstream_rq_pending_overflow, upstream_rq_timeout, and upstream_rq_retry. The exact metric names are less important than keeping the proxy’s admission state next to application RED metrics.

Retry behavior is usually configured through route policy: which failures are retryable, how many attempts are allowed, how long each attempt may run, and whether a retry budget limits the fraction of traffic spent retrying. A retry is safe only when the operation is safe to repeat or has an idempotency key. “Transient” is not a sufficient correctness argument for a write.

3. Istio attaches policy at the route boundary

An Istio VirtualService can define an HTTP retries policy with attempts, per-try timeout, and retry conditions. That makes retries easy to add consistently—and easy to apply at multiple layers accidentally. A client library, sidecar, gateway, and upstream SDK can each believe it is adding resilience while their products multiply.

Audit the entire path. If a request can be retried at four layers, the nominal “two retries” setting is not a two-attempt guarantee. Propagate an attempt number and a stable logical request ID, then make one layer the owner of the retry budget.

4. HAProxy is a hard-cap component, not an elastic promise

HAProxy can enforce connection and queue limits such as maxconn and backend queue behavior. Once a flow or connection limit is reached, the observable symptom may be latency and failed handshakes rather than high CPU. Load-balancer capacity must therefore be monitored as a first-class resource, with per-site and per-region breakdowns. A global “healthy backend count” dashboard can remain green while the front door is full.

Recovery: why rejecting traffic can be correct

GitHub reports two notable mitigations: reducing gateway retries with a code change and temporarily returning HTTP 403 for inbound Copilot Token Service requests at the load balancers, then ramping traffic back up by site. The specific status code is an API decision that depends on the caller, but the reliability principle is general: deliberate load shedding can restore a system faster than accepting work it cannot finish.

A useful recovery order is:

1. Stop multiplying work: disable or sharply reduce retries, especially on the saturated path.
2. Protect the failing dependency with admission limits, circuit breaking, or a controlled rejection response.
3. Recover a stable region or pool and verify successful completions, not just lower error rates.
4. Ramp traffic gradually while watching queue age, retry ratio, and tail latency.
5. Only then restore normal retry policy and client traffic.

Failing fast is not always user-friendly, but a bounded error is often better than a slow request that is retried five times and prevents unrelated requests from completing. The correct policy is product-specific; the capacity accounting is not.

Failure modes and security implications

• Retry multiplication: nested retry layers turn a small timeout into a large offered-load spike. Put the attempt count in traces and logs, and enforce a total deadline.
• Non-idempotent duplication: a timed-out write may have committed before the client retries it. Use idempotency keys and deduplicate at the operation boundary.
• Retry storm as denial of service: a broken or malicious client can intentionally exploit retry policy. Apply per-tenant budgets, rate limits, and authentication before expensive work.
• Metric spoofing or omission: an attacker who drives proxy queues but not application CPU can keep an autoscaler blind. Alert on admission metrics independently of app metrics.
• Regional failover amplification: moving traffic to a healthy region without suppressing retries can transfer the overload. Failover must include retry and admission policy.
• Load-shed bypass: a 403, 429, or 503 is useful only if callers respect it. Ensure SDKs do not retry the chosen overload response, or attach explicit retry guidance.
• Scraping and background work: bulk downloads, crawlers, and automation can consume the same front-door budget as interactive requests. Separate pools or quotas reduce blast radius.
• Information leakage: detailed overload responses and per-region timing can help an attacker map capacity. Expose enough telemetry to clients to behave correctly without publishing the topology.

Debugging method

1. Build the attempt timeline. Align the first latency increase, sidecar overflow, load-balancer flow saturation, autoscaler observations, retry increase, and regional traffic shift on one UTC axis.
2. Separate logical requests from attempts. Count both. If 10,000 user operations produce 70,000 internal attempts, the retry policy is part of the incident, not background noise.
3. Trace one request end to end. Carry a stable logical ID and an incrementing attempt number. Record timeout reason and remaining deadline at every hop.
4. Compare admission to completion. Plot sidecar accepted, sidecar rejected, application started, application completed, and client-visible success. A gap between accepted and started points at the proxy/gateway boundary.
5. Check the metric scope. Verify which container, route, region, and time window the HPA reads. Reproduce the desired-replica calculation from the raw metric instead of trusting a dashboard label.
6. Use a controlled load-shed experiment. In a staging environment, cap retries or reject low-priority traffic. If high-priority completion recovers while app CPU remains unchanged, you have demonstrated overload feedback.
7. Test recovery, not just failure. Kill a gateway, saturate a sidecar, delay an auth endpoint, shift a region, and cancel requests during retry backoff. Watch for retry storms and capacity leaks after the fault is removed.

Compact reproducible lab

Goal

Model the essential mismatch: four replicas have an application capacity of 500 requests per second each, but each sidecar admits only 180 attempts per second. The application metric looks cool, so the blind autoscaler does not add replicas. A sidecar-aware autoscaler sees the real gate and recovers after scaling.

Prerequisites

Python 3.9 or newer. No packages are required. This is a deliberately small discrete-time model, not a benchmark of Kubernetes or Envoy.

Exact commands

python3 - <<'PY'
from math import ceil

def simulate(label, observed_metric):
    base_requests = 1_000
    replicas = 4
    retry_queue = 0
    sidecar_limit = 180       # attempts/second/replica
    app_capacity = 500        # completed requests/second/replica
    target = 0.70

    print(f"\\n{label}")
    print("sec replicas offered admitted completed failed metric")
    for second in range(8):
        offered = base_requests + retry_queue
        sidecar_capacity = replicas * sidecar_limit
        app_capacity_total = replicas * app_capacity
        admitted = min(offered, sidecar_capacity)
        completed = min(admitted, app_capacity_total)
        failed = offered - completed
        retry_queue = min(failed, 4_000)  # one retry wave, bounded for the demo

        if observed_metric == "app":
            metric = completed / app_capacity_total
        else:
            metric = offered / sidecar_capacity

        # A simplified HPA decision every two seconds.
        if second in (1, 3, 5) and metric > target:
            replicas = min(16, max(replicas, ceil(replicas * metric / target)))

        print(f"{second:3d} {replicas:8d} {offered:7d} {admitted:7d} "
              f"{completed:9d} {failed:6d} {metric:6.2f}")

simulate("blind: HPA watches application utilization", "app")
simulate("aware: HPA watches the admission limit", "sidecar")

print("\\nretry multiplier: p=0.4, max retries=3 ->",
      sum((1 - 0.4) ** i for i in range(4)))
PY

Expected result

In the blind run, application utilization is only about 0.36 because the sidecar admits 720 attempts per second and the application can process 2,000. Replicas stay at four while the offered load grows through retries. In the aware run, the admission metric is above target, the model scales up, and the retry queue drains once sidecar capacity exceeds the offered load. The last line prints approximately 2.176, the retry-attempt multiplier for a 40% per-attempt success rate and three retries.

Revealing variations

• Set retry_queue = 0 after each iteration. The system remains overloaded but stops amplifying itself; this approximates clients honoring a rejection or retry budget.
• Change sidecar_limit to 500. The blind and aware runs now look similar because the observed bottleneck has been removed. This demonstrates why fixing the admission limit can be more valuable than adding application replicas.
• Change the scale decision to run only at seconds (3, 7). The same metric becomes less useful when its control-loop delay is longer than the overload growth time.

Primary sources

• GitHub Status: Incident with GitHub.com, August 17, 2026 — official timeline, causal chain, traffic amplification, and follow-up actions.
• The Register: GitHub blames 8-hour outage on autoscaling fail and VS Code retry storm — independent reporting and incident context, 2026-08-19.
• Kubernetes HPA controller source — metric sampling, replica calculation, stabilization, and reconciliation.
• Kubernetes Horizontal Pod Autoscaler documentation — resource, custom, and external metric behavior.
• Istio VirtualService HTTPRetry reference — attempts, per-try timeout, and retry conditions.
• Envoy router filter documentation — upstream routing, timeout, and retry behavior.
• Google SRE Book: Handling Overload — admission control, load shedding, and graceful degradation.
• Amazon Builders' Library: Timeouts, retries, and backoff with jitter — why retry policy needs bounded amplification and jitter.