Cloud platforms / AI inference deep dive

2026-09-23

Capacity-aware inference: why heterogeneous GPUs need a scheduler, not a retry loop

A model endpoint that accepts “any available GPU” sounds resilient. It is not. Different accelerators change memory capacity, tensor parallelism, batching efficiency, token latency, and sometimes the model configuration itself. The hard problem is not finding a spare machine; it is admitting work only when that machine can satisfy the request’s performance and correctness contract.

1. The concrete problem

GPU capacity is not a binary resource. A deployment may prefer an H100, tolerate an A100, and use a smaller L4 only for short prompts or a quantized model. During a capacity shortage, a naive controller does this:

create_instance(preferred_gpu)
if capacity_error:
    retry(create_instance, fallback_gpu)

That loop answers only “did the cloud allocate a VM?” It does not answer:

• Can the fallback hold the weights, runtime workspace, and KV cache?
• Does its tensor-parallel degree match the checkpoint and kernel configuration?
• Will its queueing delay violate the time-to-first-token or inter-token-latency budget?
• Can the fallback absorb scale-out traffic without starving existing requests?
• Can autoscaling later remove fallback nodes before preferred nodes?

This is increasingly visible in managed inference. AWS described capacity-aware instance pools in 2026 as ordered lists of up to five instance types, used during endpoint creation, scale-out, and scale-in; each type can have its own optimized model configuration.[1][2] That is a scheduler feature, not merely a retry policy.

2. The scheduling model

Represent each hardware option as a capacity class. A class is more than a SKU: it includes accelerator count, memory, interconnect, model artifact, runtime flags, and a measured performance envelope.

Let a request require model memory W, runtime workspace X, and a KV-cache budget K. A candidate is memory-feasible only if:

usable_memory(candidate) >= W + X + K + safety_margin

Memory feasibility is necessary, not sufficient. Define a candidate’s predicted service time as S(c, request), and its current queueing delay as Q(c). Admit the request only when:

Q(c) + S(c, request) <= remaining_deadline(request)

The scheduler therefore needs two distinct states:

• Provisioning state: which instances can be allocated, are starting, or are draining.
• Serving state: which replicas are ready, how much KV memory remains, how many sequences are active, and what the measured token rate is.

Cloud APIs expose the first state. The model server exposes the second. Joining them is the control-plane problem.

3. Control and data flow

Scale-out

1. A metric observes queue depth, pending tokens, time-to-first-token, or replica utilization.
2. The autoscaler asks for capacity, not just a number of identical machines.
3. The capacity policy tries the ordered class list, subject to quota, availability zone, and placement constraints.
4. The bootstrap process selects the matching model artifact and runtime configuration.
5. Readiness is withheld until weights are loaded, the model server is healthy, and a probe request meets a minimum performance check.
6. The router adds the replica to the eligible pool with a weight appropriate to its measured capacity.

Scale-in and recovery

When preferred capacity returns, removing fallback instances first is sensible only if their active work can drain. A safe drain sequence marks the replica non-admitting, waits for in-flight requests, migrates or expires cache state, then terminates it. AWS documents this preference ordering for its capacity-aware pools.[2]

Dynamic batching complicates the picture. A model server may delay a request briefly to form a more efficient batch; NVIDIA Triton exposes both preferred batch sizes and a maximum queue delay for this purpose.[3] A smaller fallback GPU may need a lower batch size or a stricter queue delay. Copying the preferred replica’s settings can turn “more efficient” batching into out-of-memory errors or tail-latency collapse.

Why capacity weights must be measured

If class A produces 120 tokens/s and class B produces 45 tokens/s, routing one request to each is not a balanced split. A useful first approximation weights by sustainable token throughput, then corrects for prompt length, concurrency, model configuration, and observed tail latency. Static request-count weights are often wrong for LLMs because one long generation can consume the capacity of many short requests.

4. Implementation details

A practical implementation can be built from five small records. The names differ across projects, but the responsibilities recur.

The main functions are similarly separable:

• feasible(class, request, replica) checks memory, model revision, quota, and protocol compatibility.
• score(candidate, request) estimates queue delay, service time, cost, and risk.
• admit(request, decision) atomically reserves queue or KV capacity.
• reconcile_capacity() turns desired class counts into cloud allocation operations.
• reconcile_replicas() turns instance lifecycle into Ready/Draining/Failed serving membership.
• route(request) chooses among Ready replicas; it should not call the cloud API synchronously on the request path.

Kubernetes scheduling makes a useful analogy: placement constraints and affinity decide where a pod can run, while the workload itself still needs readiness and resource accounting.[4] For inference, GPU type, model revision, and runtime flags behave like a stronger form of node affinity. A pod being scheduled is not equivalent to an inference replica being ready.

5. Tradeoffs and failure modes

Overload control matters more than perfect placement. Google’s SRE guidance treats throttling, queue limits, load shedding, and degraded responses as tools for preventing cascading failure.[5] An inference scheduler should expose a deliberate policy for “no class can satisfy the contract”: reject early, use a smaller model, switch to asynchronous work, or return a cached answer. Waiting forever is not graceful degradation.

Common bugs

• Capacity thrash: the controller repeatedly asks for an unavailable preferred type, never advancing its backoff or fallback state.
• Cold-start stampede: every queue alarm launches a new model container whose weights compete for the same network and storage bandwidth.
• Configuration mismatch: fallback hardware loads weights but fails later because tensor parallelism or quantization metadata is wrong.
• False readiness: an HTTP health endpoint turns green before the model has usable GPU memory or a warmed kernel cache.
• Return-to-preferred storm: recovery immediately drains fallback capacity and overloads preferred replicas.
• Accounting blindness: cost and throughput are measured per instance, hiding that a “cheap” fallback consumes three times the GPU-seconds per request.

6. Debugging production behavior

When a fallback incident occurs, build one timeline that joins the cloud control plane and the inference data plane.

1. For each capacity request, record class, zone, provider error, backoff, and policy version.
2. For each replica, record boot start, image ready, weights ready, model-ready probe, and first admitted request.
3. For each request, record selected class, queue wait, prefill time, decode time, generated tokens, finish reason, and deadline remaining.
4. Compare request-count share with token-count share. LLM load is often token-shaped.
5. Plot KV-cache utilization and queue depth by class. A fallback may have idle GPU compute while being memory-bound.
6. Check whether a scale-out event caused its own outage through image pulls, weight downloads, or rate-limit exhaustion.
7. During recovery, verify that draining replicas stop receiving new requests before they disappear from the router.

Useful metrics include capacity_attempts_total{class,outcome}, replica_ready_seconds{class}, request_admission_total{class,decision}, queue_wait_seconds{class}, tokens_per_second{class}, kv_blocks_free{replica}, fallback_share, and deadline_exhaustion_total. Alert on fallback share and deadline exhaustion, not just instance count.

7. Reproducible lab

Goal: observe why request-count balancing overloads a slower fallback, then compare it with token-weighted routing and admission control.

Prerequisites: Python 3.11 or newer. No third-party packages.

mkdir -p /tmp/capacity-aware-lab
cd /tmp/capacity-aware-lab
cat > simulate.py <<'PY'
from dataclasses import dataclass

@dataclass
class Class:
    name: str
    tokens_per_second: float
    token_budget: float = 0
    rejected: int = 0

requests = [500, 500, 500, 500, 500, 500, 500, 500, 500, 500,
            2000, 2000, 2000, 2000, 2000]
classes = [Class("preferred", 120), Class("fallback", 45)]

# Naive policy: alternate by request count.
for i, tokens in enumerate(requests):
    c = classes[i % 2]
    c.token_budget += tokens
print("request-count routing:", [(c.name, round(c.token_budget / c.tokens_per_second, 1))
                                  for c in classes], "seconds of work")

# Capacity-aware policy: send to the class with the least projected work.
classes = [Class("preferred", 120), Class("fallback", 45)]
for tokens in requests:
    feasible = [c for c in classes if (c.token_budget + tokens) / c.tokens_per_second < 30]
    if not feasible:
        classes[0].rejected += 1
        continue
    c = min(feasible, key=lambda x: (x.token_budget + tokens) / x.tokens_per_second)
    c.token_budget += tokens
print("capacity-aware routing:", [(c.name, round(c.token_budget / c.tokens_per_second, 1), c.rejected)
                                  for c in classes])
PY
python3 simulate.py

Expected result: request-count routing gives the slow fallback the same number of requests as the preferred class, even though it has much less token capacity. The capacity-aware policy assigns work using projected service time and rejects only after both classes exceed the explicit 30-second queue budget. The exact numbers are deterministic; the point is the policy difference.

Variation 1: change the final five requests from 2,000 to 10,000 tokens. Observe that token-aware routing rejects earlier instead of allowing one long generation to poison tail latency.

Variation 2: add a third class at 20 tokens/s with a different model revision. First allow it for all requests, then make feasible() reject requests whose required revision does not match. This demonstrates why “has GPU memory” is not a complete compatibility check.

Lab limitation: this is a queueing model, not a GPU benchmark. Real systems also have batching, prefill/decode asymmetry, memory fragmentation, kernel warm-up, network transfer, and admission races.

8. Security and operational design

Heterogeneous fallback changes the execution boundary. A smaller class may use a different container, model artifact, driver, node image, availability zone, or data path. Sign and verify every image and model artifact; bind the approved digest and model revision to the capacity-class record instead of trusting a mutable tag.

Keep policy explicit. A request containing sensitive data may be allowed on the preferred private class but forbidden on a fallback in another zone or account. Likewise, a tenant’s priority should not silently become a license to consume every scarce preferred GPU. Use per-tenant quotas, queue limits, and auditable placement decisions.

Finally, test the transitions: preferred capacity unavailable, fallback partially available, model download slow, fallback slower than its benchmark, and preferred capacity returning while fallback queues are full. A good game day verifies that the system fails closed on incompatibility, sheds load before memory exhaustion, and drains without duplicating side effects.

Takeaway

Heterogeneous GPUs are useful precisely because they are different. That difference makes a retry loop insufficient. Build a capacity class with a reproducible model/runtime configuration, keep provisioning state separate from serving readiness, route by projected token work rather than request count, preserve deadlines, cap queues, and drain fallback capacity deliberately. The winning design is not the one that finds any GPU; it is the one that can explain why this request was admitted to this replica and still met its contract.