Cloud platforms / reliability deep dive

2026-09-22

AI provider failover: why three model vendors can fail together

“Fail over from Provider A to Provider B” sounds like redundancy. It is only redundancy if the two paths differ in the failure domains that matter. A September 3, 2026 outage showed the uncomfortable version: ChatGPT, Claude, and Grok degraded in the same morning, while the public record pointed to different immediate causes rather than one confirmed shared outage. The engineering lesson is not that the vendors secretly share one machine. It is that service-level diversity is not the same thing as dependency diversity.

1. The concrete problem

Suppose an application uses one model provider for summarization and keeps a second provider configured as a backup. The primary returns timeouts. The client tries the backup. The backup also times out. The team concludes that “AI providers are unreliable,” but that sentence is too coarse to guide a fix.

There are at least four different failure shapes:

1. Provider-local: one model deployment, account, or API gateway is unhealthy.
2. Shared infrastructure: multiple vendors depend on the same cloud region, network, compute partner, identity service, or egress path.
3. Client-induced: the failover policy sends a retry storm to the backup and turns a recoverable fault into a second outage.
4. Semantic: the backup responds successfully but has incompatible context limits, tool behavior, safety policy, or output schema.

Only the first shape is fixed by swapping vendors. The second requires dependency mapping; the third requires admission control; the fourth requires contract testing and graceful degradation.

2. What the September outage actually establishes

On September 3, OpenAI’s status page recorded elevated errors across ChatGPT and Codex. It lists 15 affected ChatGPT components and four Codex components, with the incident beginning at 10:43 UTC, mitigation at 11:17 UTC, and resolution at 12:55 UTC.[1] OpenAI’s later write-up attributed its incident to a routing configuration change that was interpreted incorrectly; returning traffic then overloaded some recovering services.[2]

Anthropic’s incident record independently shows elevated errors for multiple Claude models from 13:26 UTC, cause identified at 13:41, and impact ending at 16:16 UTC.[3] WIRED reported that xAI described the Grok outage as an outage at its Memphis compute center and that Anthropic and xAI had announced a compute partnership with SpaceX.[4]

Those facts justify a reliability lesson about correlated exposure. They do not prove that one Azure failure took down all three providers. WIRED reported that Microsoft Azure, AWS, and Cloudflare did not report a corresponding outage, and the vendors did not publish one shared root cause.[4] Treating a timing correlation as a confirmed infrastructure cause is exactly how dependency analysis becomes folklore.

3. Common-mode failure and dependency graphs

A useful model represents a request path as a graph rather than a vendor list. Nodes are components; edges are calls, network paths, credentials, or operational dependencies. A failure domain is any set of nodes likely to fail together because they share a physical, administrative, software, or economic boundary.

For a request to succeed, every mandatory edge on the chosen path must work. For a failover to add availability, the two paths should have low overlap in the failure domains that dominate the service’s risk. Let F(A) and F(B) be the sets of failure domains that can break providers A and B. The useful part of diversity is not the size of either set; it is the small intersection F(A) ∩ F(B).

Independence is never absolute. Two providers may share an internet exchange, a certificate authority, a DNS resolver, a cloud region, a GPU supplier, a model safety dependency, or your own client library. The practical goal is to identify the intersections that are both plausible and high impact, then decide which ones deserve mitigation.

Why model serving makes this harder

Traditional web failover usually moves traffic between interchangeable CPU replicas. Model inference is less interchangeable: specialized accelerators are capacity-constrained, model weights are large, warm-up is expensive, and output behavior is part of the application contract. A cold backup may be reachable but unable to absorb the primary’s traffic. A warm backup may be cheaper because it is smaller, but then quality or context length changes.

4. Control flow of a useful failover

A robust client treats failover as a bounded state machine, not a second URL inside a catch block.

The state machine needs five explicit decisions:

1. Classify: retry connection resets, selected timeouts, and provider-declared overload; do not retry invalid credentials, malformed input, or policy refusal.
2. Budget: carry one deadline through all attempts. If the primary uses 6 seconds of an 8-second budget, a 30-second backup timeout is a bug.
3. Admit: use a circuit breaker, concurrency limit, and token or cost budget before sending traffic to the backup.
4. Preserve identity: attach an idempotency key and request ID so a timed-out generation cannot be duplicated blindly.
5. Validate: check schema, tool-call shape, citations if required, maximum length, and safety policy before returning backup output.

OpenAI’s incident write-up is a compact example of why recovery matters after the initial fault: restoring routes caused returning traffic to overload some services. Failover systems have the same property at smaller scale. Recovery traffic is a workload spike, not a free side effect.[2]

5. Design tradeoffs and failure modes

Retry storms

If N clients each make r attempts, the provider sees approximately N × r requests in the failure window, before considering exponential backoff. If the primary fails after accepting work, retrying may create duplicate generations or duplicate tool calls. Backoff with jitter reduces synchronization, but it does not replace a concurrency cap or a circuit breaker.

Semantic split-brain

HTTP success is not application success. Provider A may return a tool call with strict JSON while Provider B returns prose; one may support a 200k-token context while another truncates; one may refuse a request that the other answers. Keep an adapter contract and run compatibility tests against real provider responses. The fallback path must be observable as a distinct route, model, and policy decision.

6. Debugging the dependency chain

During an incident, do not begin with “which provider is down?” Begin with a time-aligned matrix.

1. Record client-side DNS, TCP, TLS, HTTP status, provider request ID, model, region hint, and elapsed time.
2. Separate connection failure, admission rejection, quota response, model execution timeout, malformed output, and safety refusal.
3. Plot primary and backup attempts on one UTC timeline, including circuit-breaker state and concurrency.
4. Compare direct provider probes with the production path. A direct probe can succeed while your proxy, egress, DNS, or identity layer is broken.
5. Check whether the backup’s error rate rose only after failover began. That pattern indicates induced overload, not necessarily a shared upstream fault.
6. Inspect control-plane changes: routing rules, credentials, model aliases, feature flags, DNS, quota, and rollout percentage.
7. Preserve uncertainty. Mark “confirmed,” “reported,” and “inferred” separately in the incident timeline.

Useful metrics are provider_attempts_total by provider and outcome, failover_rate, time_to_first_token, deadline_exhaustion_total, backup_capacity_headroom, and semantic_validation_failures. Aggregate success rate hides the exact problem: a system can keep 99.9% success while using its backup for 90% of traffic and silently paying more.

7. Reproducible lab

Goal: show why a backup that is healthy in isolation can fail when a primary outage redirects too much traffic, then add bounded admission.

Prerequisites: Python 3.11 or newer. No third-party packages.

mkdir -p /tmp/failover-lab
cd /tmp/failover-lab
cat > simulate.py <<'PY'
from dataclasses import dataclass

@dataclass
class Provider:
    name: str
    capacity: int
    healthy: bool = True
    accepted: int = 0
    rejected: int = 0

def run(primary_healthy: bool, guarded: bool):
    a = Provider("A", capacity=70, healthy=primary_healthy)
    b = Provider("B", capacity=70)
    requests = 100
    for _ in range(requests):
        target = a if a.healthy and a.accepted < a.capacity else b
        if guarded and target is b and b.accepted >= 40:
            target.rejected += 1
        elif target.accepted < target.capacity:
            target.accepted += 1
        else:
            target.rejected += 1
    print({p.name: {"accepted": p.accepted, "rejected": p.rejected}
           for p in (a, b)}, "guarded", guarded)

run(primary_healthy=True, guarded=False)
run(primary_healthy=False, guarded=False)
run(primary_healthy=False, guarded=True)
PY
python3 simulate.py

Expected result: with A healthy, 70 requests go to A and 30 to B. When A fails, B receives the redirected load but can accept only 70, so 30 requests fail. With the guard enabled, B is deliberately capped at 40 and the remaining requests are rejected or degraded earlier. That looks worse in raw acceptance, but it prevents the backup from being driven into an uncontrolled overload regime.

Variation 1: replace the hard cap with a token bucket and add 0–200 ms of random jitter before retries. Observe that jitter changes synchronization, while the cap changes the maximum load.

Variation 2: make B “accept” all requests but return a different JSON shape. Add a schema check after the provider call. Availability without semantic validation is a false success.

Lab limitation: the model is deterministic and treats every request as equal cost. Real inference has token-dependent service time, queueing, batching, rate limits, and provider-specific error semantics.

8. Production and security implications

Map dependencies at the level your risk actually lives. “Provider B” is not a sufficient inventory entry. Record account, region, ingress, network path, identity provider, DNS, compute partner where known, model family, quota, data-retention policy, and operational ownership. Revisit the map after every architecture or procurement change.

Failover also changes the security boundary. A backup may process data in another jurisdiction, retain prompts under different terms, or expose different tools. Do not copy credentials and tool permissions blindly. Use provider-specific least-privilege credentials, redact or transform sensitive inputs before cross-provider fallback, and make a policy decision before sending regulated data to a secondary path.

Prompt injection and tool calls make retries especially dangerous. A timed-out request may have completed a side effect even if the client saw no response. Require idempotency keys for external actions, separate generation from execution, and put human or policy approval in front of irreversible operations. A fallback model should not gain broader tool access merely because the primary is unavailable.

Test failover as a game day, not a configuration checkbox. Disable the primary in a controlled environment, inject latency rather than only hard errors, cap backup concurrency, and measure whether the product degrades into a useful mode. Include recovery: restore the primary while queues are full and verify that traffic drains without another overload.

Takeaway

The September 3 outage is valuable precisely because the public evidence does not support one tidy shared-cause story. OpenAI documented a routing fault; Anthropic documented a multi-model infrastructure incident; xAI’s parent described a Memphis compute-center outage. The simultaneous customer experience still exposed a design weakness: application teams often buy “provider diversity” without measuring dependency overlap. Build a graph, budget the deadline, cap failover load, validate semantics, isolate credentials, and test recovery. A backup that cannot survive the primary’s traffic is not redundancy. It is a second queue waiting for the same incident.