Cloud-native / API control plane deep dive

Kubernetes API Priority and Fairness: the queues that keep one tenant from starving the control plane

A Kubernetes API server can be healthy at the process level and still be unusable: one controller, user, or automation loop can consume every worker while the requests that would repair the cluster wait behind it. API Priority and Fairness (APF) turns overload into a scheduling problem with priority levels, concurrency “seats,” per-flow queues, and shuffle sharding.

Cloud-native / API control plane deep dive · FlowSchema, PriorityLevelConfiguration, queue-set internals, fairness, failure modes, and a reproducible lab

2026-09-03

The concrete problem

The Kubernetes API server is the control-plane rendezvous point. Controllers read objects, write status, renew leases, create workloads, and watch changes through it. During a failure or a rollout, that traffic can increase exactly when operators need the API most.

Before APF, the main protection was a pair of global in-flight limits: one for mutating requests and one for non-mutating requests. Those limits stop total concurrency from growing without bound, but they do not answer the important question: which request should get the next slot? A noisy client can fill the available slots and force unrelated clients to receive HTTP 429 responses or wait behind a queue that knows nothing about tenant identity.

APF does not make an expensive API call cheap, and it does not guarantee that every request succeeds. Its narrower job is to preserve a controlled amount of concurrency and prevent one classified flow from monopolizing the waiting room of an entire priority level.

The two objects that define policy

APF separates classification from capacity and queuing. A FlowSchema says which requests belong together and points at a PriorityLevelConfiguration (PLC). The PLC says whether that class is exempt or limited, and, for a limited class, how many queues and seats it gets.

The distinction between APF and authorization matters in production. A request can be classified into a high-priority flow and then be rejected by RBAC. Conversely, a request that is authorized can still wait or receive 429 because APF is protecting the server’s capacity.

From HTTP request to execution

The request path has a small but important sequence. The filter extracts request information and user identity, the APF controller matches a FlowSchema, and the priority level either dispatches immediately or hands the request to its queue set.

“Seats” are the unit APF uses for concurrency. A simple request generally consumes one seat, while a large LIST can be estimated as more work. The work estimator lets the queue set account for seat demand rather than treating every request as identical. Long-running requests such as watches have special handling and must not be casually treated as ordinary short requests when sizing a level.

The default configuration contains an exempt level for critical traffic and several limited levels for ordinary traffic. Exempt means no APF queue and no APF execution limit for that level; it is a deliberately powerful escape hatch and should not be copied for tenant workloads.

Why shuffle sharding works

A single FIFO queue gives good ordering but poor isolation: one elephant flow can fill it. Giving every flow a private queue is expensive and can create thousands of mostly empty queues. APF uses shuffle sharding as a middle ground.

For each flow, the API server hashes a stable flow identifier and deals a hand of distinct queue indices from the priority level’s queue set. The request goes to the shortest queue in that hand. Two flows usually have different hands, so a heavy flow tends to occupy only a small subset of queues. A victim flow is harmed only when its hand overlaps the heavy flow’s hand and the shared candidates are the best choices.

queues = [0, 1, 2, 3, 4, 5, 6, 7]
handSize = 2

hash(flow="service-a") → hand [1, 6] → queue 6 (shorter)
hash(flow="service-b") → hand [0, 6] → queue 0 (shorter)
hash(flow="service-c") → hand [2, 5] → queue 5

If service-a becomes an elephant, service-b still has queue 0 as an escape.
It is not perfect isolation: the hands overlap at queue 6.

The implementation uses a shuffle-sharding dealer. In the queue set, shuffleShardLocked calls DealIntoHand, examines candidate queue work in a rotated order, and selects the candidate with the smallest queued seat-seconds. That last detail is important: the shortest queue is measured in estimated work, not merely in request count.

There is a probabilistic tradeoff. Too few queues create collisions; too small a hand gives a flow little ability to avoid a busy candidate; too large a hand makes many flows overlap and weakens isolation. The right values depend on the number of independent flows and the concurrency budget, not on a universal “recommended” integer.

Concurrency shares, lending, and borrowing

Suppose a server’s effective concurrency budget is 600 seats and the limited priority levels have shares 30, 10, and 60. Ignoring rounding and dynamic adjustments, their nominal limits are approximately:

sum shares = 30 + 10 + 60 = 100
level A = ceil(600 × 30 / 100) = 180 seats
level B = ceil(600 × 10 / 100) =  60 seats
level C = ceil(600 × 60 / 100) = 360 seats

Nominal is not necessarily current. Kubernetes measures recent seat demand and can adjust a level’s current concurrency limit. A level can lend unused capacity, and another can borrow subject to configuration. The point is to avoid permanently marooning idle capacity while preserving a lower bound for important levels.

This is why a PLC with nominalConcurrencyShares: 10 is not a hard cap of ten requests. It participates in a global allocation algorithm, and a request may consume more than one seat. If a team needs a hard external quota, it should use the relevant API quota and workload controls as well; APF is a server-scheduling policy.

Open-source implementation landmarks

The code is intentionally split into a request filter, a configuration controller, and a generic queue-set implementation. Reading it in that order makes the control flow easier to follow than starting in the queue math.

One subtle implementation invariant is the two-phase configuration change. Queueing configuration is constructed first; dispatching configuration supplies the concurrency allocation afterward. Reconfiguration does not casually discard non-empty queues, because dropping queued requests would turn a policy edit into an unexplained client failure.

Failure modes and security implications

• Wrong FlowSchema precedence. A custom schema with a numerically smaller matchingPrecedence can capture traffic intended for a built-in class. Inspect the matched schema and PLC in APF debug output before changing shares.
• One flow by accident. An absent distinguisher collapses every matching request into one flow. That may be the intended global FIFO, but it defeats per-user or per-namespace isolation.
• Too many exempt requests. Exemption bypasses the queue and the APF limit. A broad exempt rule lets an authenticated client compete directly with control-plane traffic and can recreate the original overload problem.
• Queue full is policy, not proof of compromise. APF returns 429 when the selected queue is full under the configured conditions. Clients must use bounded exponential backoff with jitter; synchronized immediate retries can turn a useful queue into a retry storm.
• Fairness is per apiserver. Each API server has local queue state. A load balancer can send successive requests from the same client to different servers, so fairness and debugging views are not one global queue.
• Classification is not authorization. FlowSchema rules do not grant access. Keep RBAC narrow and review APF changes separately; using APF as a security boundary is a category error.
• Expensive work can dominate seats. A request that takes a long time or is estimated with high seat demand can occupy capacity even when the queue is fair. Fix the API usage pattern, pagination, caching, or controller behavior instead of only increasing shares.

Debugging a congested API server

1. Separate process saturation from APF waiting. Inspect CPU, memory, storage latency, request duration, and the API server’s APF metrics. A queue can be behaving correctly while the backend is the real bottleneck.
2. Find the class. Inspect FlowSchema and PriorityLevelConfiguration objects, especially precedence, subjects, resource rules, distinguisher, queues, and shares.
3. Use the debug endpoints. On a secured, local admin path, query /debug/api_priority_and_fairness/dump_priority_levels and /debug/api_priority_and_fairness/dump_requests. Treat dumps as diagnostic snapshots: the queue-set dump stops the queue set while it is collected.
4. Read metrics by phase. Watch request counts and seats for waiting, executing, and dispatched phases. A rising waiting count with stable execution suggests queue pressure; a rising execution duration suggests backend work.
5. Compare identities and destinations. Check whether the problem follows one service account, namespace, or API server. That distinguishes a flow collision from a global capacity problem.
6. Change one variable. Temporarily widen queue count or correct a distinguisher in a controlled environment. Do not “fix” an incident by making every request exempt.

Compact lab: see classification and queue policy

This lab uses a disposable Kubernetes cluster and the stable flowcontrol.apiserver.k8s.io/v1 API. It creates a narrow schema for GET/LIST requests for Pods in one namespace. It does not grant permission; use an identity that already has access, or observe the classification even if the final request is denied by RBAC.

Goal and prerequisites

Have kubectl, jq, and a Kubernetes cluster with APF enabled (Kubernetes 1.29 or newer is a practical baseline). Use a disposable cluster or save the existing FlowSchema/PLC objects first. APF objects are cluster-scoped and a mistaken precedence can affect other traffic.

Apply a narrow policy

kubectl create namespace apf-lab

kubectl apply -f - <<'YAML'
apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: PriorityLevelConfiguration
metadata:
  name: apf-lab-limited
spec:
  type: Limited
  limited:
    nominalConcurrencyShares: 10
    limitResponse:
      type: Queue
      queuing:
        queues: 4
        handSize: 2
        queueLengthLimit: 8
---
apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: FlowSchema
metadata:
  name: apf-lab-pods
spec:
  matchingPrecedence: 900
  distinguisherMethod:
    type: ByUser
  priorityLevelConfiguration:
    name: apf-lab-limited
  rules:
  - subjects:
    - kind: Group
      group:
        name: system:authenticated
    resourceRules:
    - verbs: ["get", "list"]
      apiGroups: [""]
      resources: ["pods"]
      namespaces: ["apf-lab"]
YAML

kubectl get prioritylevelconfiguration apf-lab-limited -o yaml
kubectl get flowschema apf-lab-pods -o yaml
kubectl get --raw='/debug/api_priority_and_fairness/dump_priority_levels' | jq .

Generate work and observe

for i in $(seq 1 40); do
  kubectl get pods -n apf-lab --ignore-not-found &
done
wait

kubectl get --raw='/metrics' | grep -E 'apiserver_flowcontrol|apiserver_current_inflight_requests'
kubectl get --raw='/debug/api_priority_and_fairness/dump_requests' | jq .

Expected result: the objects show a limited level with four queues and a two-queue hand. If the API server is not busy, the burst may complete without visible waiting; that is a valid result, because APF only queues when the level’s current seats are occupied. Under contention, the dump and metrics expose waiting requests and the selected priority level. A 403 means RBAC denied the operation after classification; it does not prove that the FlowSchema failed.

Revealing variations

# Make every flow inspect only one candidate queue.
kubectl patch prioritylevelconfiguration apf-lab-limited --type merge \
  -p '{"spec":{"limited":{"limitResponse":{"queuing":{"handSize":1}}}}}'

# Increase the queue population and candidate hand again.
kubectl patch prioritylevelconfiguration apf-lab-limited --type merge \
  -p '{"spec":{"limited":{"limitResponse":{"queuing":{"queues":16,"handSize":4}}}}}'

# Remove the experiment.
kubectl delete flowschema apf-lab-pods
kubectl delete prioritylevelconfiguration apf-lab-limited
kubectl delete namespace apf-lab

With handSize: 1, collisions are more damaging because a flow has no alternate candidate. With more queues and a larger hand, isolation usually improves, but memory, bookkeeping, and overlap also change. The exact queue dump depends on the server’s concurrent load and on which identity the client presents.

The production lesson

APF is a control-plane admission scheduler, not merely a rate limiter. Its useful unit is the flow: a classified slice of requests with a distinguishable identity. Its useful protection is not “everyone gets the same number of requests,” but “a heavy flow has bounded blast radius, and important classes retain a path to execution.”

When designing a custom policy, start from the failure you want to contain: a controller, namespace, service account, or API surface. Then verify the full path—classification, distinguisher, nominal shares, current allocation, queue collision, client retry behavior, and backend latency. If any one of those is guessed, the YAML may look protective while the production queue behaves like one large FIFO again.