Cloud-native infrastructure / control-plane deep dive

Kubernetes watches: resourceVersion, bookmarks, and the cache behind controllers

A Kubernetes controller does not poll the whole cluster every second. It performs a snapshot read, opens a change stream, and reconciles from an eventually changing view. The hard part is not opening a TCP connection. It is resuming that stream without missing an event, overloading the API server, or pretending that a stale cache is current.

Cloud-native infrastructure deep dive · Kubernetes API machinery, LIST/WATCH, resource versions, watch cache, relists, and controller correctness

2026-08-21

The concrete problem

Suppose a controller maintains one Deployment per custom resource. It needs to answer two questions continuously:

What objects exist right now?
What changed since the last state I processed?

A naïve loop lists all objects, sleeps, and lists again. That wastes bandwidth, adds read load, and creates a race between the end of one list and the beginning of the next. A naïve “watch only” loop has the opposite problem: it has no initial snapshot, and it cannot know what happened before it connected.

Kubernetes combines the operations:

controller kube-apiserver storage │ │ │ │ LIST → objects + snapshot RV │ │ │◄─────────────────────────────────────│◄──── read current state ──│ │ │ │ │ WATCH from snapshot RV │ │ │─────────────────────────────────────►│ │ │ ADDED / MODIFIED / DELETED │ watch cache + backend │ │◄─────────────────────────────────────│◄──────────────────────────│ │ │ │ │ BOOKMARK: current progress │ │ │◄─────────────────────────────────────│ │ │ │ │ │ 410 Gone → relist, then re-watch │ │ │◄─────────────────────────────────────│ │

The key is metadata.resourceVersion. It is an opaque position in the API server's resource history. A controller should store and replay it, but should not parse it, compare it as a timestamp, or assume it is globally sequential across resource types.

Core invariant: a watch is valid only relative to a resource-version position. If that position has fallen out of the server's retained history, the correct recovery is a fresh LIST followed by a new WATCH—not an invented attempt to continue from an old cursor.

LIST and WATCH are one protocol, not two unrelated calls

A list response contains objects and a collection-level metadata.resourceVersion. Conceptually, that version says: “this snapshot represents the state known at position R.” The controller then asks for changes starting at R. With the usual watch semantics, the server sends changes after that position, so the controller does not double-apply the snapshot's objects as new events.

# Snapshot, and print the collection resourceVersion
kubectl get configmaps -A -o jsonpath='{.metadata.resourceVersion}{"
"}'

# Start a stream from a known position
kubectl get configmap demo --watch --resource-version=12345

The exact server behavior depends on the requested version and watch options, so production controllers use the client libraries rather than hand-assembling URLs. The conceptual contract remains stable:

Operation	Input	Output	Correctness role
LIST	selector, consistency/resource-version options	objects plus a collection resourceVersion	establishes a baseline snapshot
WATCH	resourceVersion and selectors	event stream: ADDED, MODIFIED, DELETED, BOOKMARK, ERROR	moves the local view forward
BOOKMARK	`allowWatchBookmarks=true`	progress marker, usually no object payload	proves the server has advanced past a position
ERROR	often an API status object	for example HTTP 410 / “too old resource version”	tells the client its cursor cannot be resumed

“Starting at version R” is deliberately not the same as “give me every event ever.” The server retains a bounded history, and storage compaction or cache eviction can remove old positions. Kubernetes controllers are therefore designed around convergence: after a relist, the controller recomputes desired state from the new snapshot.

What a controller actually does

The common client-go path is a pipeline of deliberately boring components. The boringness is a feature: each layer has one ownership boundary.

HTTP LIST/WATCH │ ▼ Reflector ─── maintains a stream and retries/re-lists when necessary │ ▼ DeltaFIFO / queue ─── records additions, updates, deletes, relist deltas │ ▼ Indexer / Store ─── local keyed snapshot used by reconciliation │ ▼ workqueue ─── coalesces keys and schedules retries │ ▼ Reconciler ─── reads current objects, computes desired state, writes changes

Reflector: the cursor owner

In client-go/tools/cache/reflector.go, Reflector runs the list/watch loop. The important behavior is not its method names but its recovery policy:

List objects and obtain the list's resourceVersion.
Replace or reconcile the local store with that snapshot.
Start a watch from the resulting resourceVersion.
For each event, update the store and enqueue work.
On a closed stream, transport error, or expired history, retry or relist according to the error.

Older and newer client-go releases have changed implementation details, including streaming list support and watch-list behavior. Read the version of the source deployed with your controller. The stable design obligation is to make a relist safe and make reconciliation idempotent.

Indexer: a cache, not a source of truth

The local indexer is optimized for reads such as “find all Pods with this label.” It is not a second authoritative database. Its contents may lag the API server, and a crash erases it. A reconciler should use it to avoid API traffic, then tolerate a not-found, conflict, or stale read by requeueing and rereading as appropriate.

Workqueue: why events are hints

Several updates to the same object may collapse into one queue key. A queue item therefore means “reconcile this key,” not “apply exactly this historical event.” The reconciler fetches the latest object and computes the desired result. That is why missed intermediate updates do not necessarily break correctness: the desired state is derived from the current state.

Design rule: event handlers should enqueue identities; reconciliation should read state and make an idempotent decision. Code that treats an event payload as a durable command is using WATCH for a job it was not designed to do.

The API server watch cache

Serving every controller's watch directly from etcd would turn a busy cluster into a large number of backend streams and repeated decoding work. The kube-apiserver therefore commonly inserts a watch cache between storage and clients.

etcd / storage history │ one storage watch, decoded objects ▼ cacher / watch cache ├── bounded ring of recent events ├── current object state for list responses └── fan-out to many API clients │ ├── controller A watch ├── controller B watch └── kubectl watch

The cache has two useful properties:

Fan-out: many clients can receive the same change without each creating a backend watch.
Short replay: a client that reconnects while its version is still in the ring can resume without a full storage read.

It also has a hard boundary. The ring is finite. If a client is disconnected for too long, the requested position is older than the oldest retained event. The server cannot safely manufacture the missing sequence, so it returns an expired-history error, commonly HTTP 410. This is not an exceptional bug in the cache; it is the protocol telling the client to rebuild its view.

Bookmarks: progress without object churn

A long-lived watch may be quiet. A controller can be healthy while no matching object changes for minutes, but a load balancer, proxy, or client still needs evidence that the stream is alive and that the server has advanced. A BOOKMARK event carries a resourceVersion without repeating an object.

Bookmarks are progress hints, not data events:

Do not update an object from a bookmark; it has no object state to apply.
Do record the bookmark position if your client library's contract permits it and you understand the restart semantics.
Do not expect a bookmark at an exact time or exact interval; the server may send them opportunistically.
Do not use a bookmark to skip a required relist after an expired watch.

At the wire level, a watch event is a JSON envelope with a type and an object. For normal events, the object is the changed Kubernetes resource. For a bookmark, the object is a partial metadata object carrying the progress version. This small distinction is a common source of decoder bugs in custom clients.

Compaction and the 410 recovery path

There are two kinds of “old” position to keep separate:

Situation	What the client sees	What to do
Watch socket failed, cursor still retained	network or EOF error	reconnect from the last safe version, with backoff and jitter
Cursor is outside retained history	HTTP 410 / StatusReasonExpired	LIST a fresh snapshot, replace local state, then WATCH from its version
Object disappeared during reconciliation	404 / NotFound	treat deletion as a valid current state and clean up dependents if needed
Concurrent writer changed the object	409 / Conflict	reread, recompute, and retry the mutation with a bounded policy

The dangerous implementation is “on 410, keep retrying the same watch.” That can create a tight failure loop and never repair the local cache. The safe implementation makes relist a first-class transition in the state machine.

watching(R)
  ├─ event → apply, enqueue, watching(newer R)
  ├─ EOF/network error → backoff, watching(last-safe R)
  ├─ 410 Expired → LIST → replace local view → watching(snapshot R)
  └─ auth/selector/serialization error → surface or stop; do not spin

Consistency and version choices

Kubernetes exposes resource-version options because “latest” has a cost and a meaning. The API concepts documentation describes consistency choices such as an empty resourceVersion, an exact value, or a value that means “at least as fresh as.” Do not cargo-cult a flag: choose it based on what the controller needs.

Choice	Use when	Tradeoff
ordinary LIST	you need a current baseline under normal API semantics	may involve a storage read and can be expensive for large collections
watch from an exact version	you are resuming a known cursor	fails if the cursor has expired; that failure is correctness-preserving
cache-served or “not older than” reads	you can tolerate cache latency and want to reduce backend pressure	the view may not include the newest write yet
streaming list / watch-list where supported	large collections make list payload and startup cost significant	version/client compatibility and implementation complexity increase

For most controllers, the most important practical consistency guarantee is not “every read is linearizable.” It is: the controller eventually observes a state that is fresh enough, and its reconciler converges when observations arrive out of order or are repeated.

Compact reproducible lab

Goal: observe normal watch events, see resourceVersion advance, and distinguish a bookmark from a resource mutation.

Prerequisites: kubectl and access to a disposable Kubernetes cluster. The commands use only a temporary namespace. A local kind or minikube cluster works; no cloud account is needed.

# 1. Create a disposable object
ns=watch-lab
kubectl create namespace "$ns"
kubectl -n "$ns" create configmap demo --from-literal=version=one

# 2. Capture the current object version
rv=$(kubectl -n "$ns" get configmap demo -o jsonpath='{.metadata.resourceVersion}')
printf 'starting resourceVersion=%s\n' "$rv"

# 3. In one terminal, watch from that position.
#    --output-watch-events exposes the envelope type.
kubectl -n "$ns" get configmap demo \
  --watch --output-watch-events --allow-watch-bookmarks \
  --resource-version="$rv"

# 4. In another terminal, mutate and then delete the object.
kubectl -n "$ns" patch configmap demo --type merge -p '{"data":{"version":"two"}}'
kubectl -n "$ns" delete configmap demo

# 5. Cleanup after stopping the watch
kubectl delete namespace "$ns"

Expected observations: the stream reports a MODIFIED event whose object has a newer resourceVersion, then a DELETED event. A BOOKMARK may appear between changes or during an otherwise quiet stream; its object is metadata progress rather than a full ConfigMap. Exact timing is intentionally not guaranteed.

Variation 1: start a watch with --resource-version=0. This asks the API machinery for a cache-appropriate starting point rather than an exact historical cursor; compare startup behavior with an exact version.

Variation 2: stop the watch, create several updates, and resume with an old captured version. On a small, quiet cluster it may still work because the history is retained. To observe expiration deterministically requires a deliberately configured short etcd/watch-cache history or a busy test cluster; do not create artificial churn on a shared production cluster just to force a 410.

If the lab fails: kubectl may be installed without a current context, producing “connection refused” or “no configuration.” Run kubectl config get-contexts, point it at a disposable local cluster, or use the wire-level fallback below. Do not paste credentials or a production kubeconfig into a public lab.

Wire-level fallback: against any authorized test API server, run kubectl -v=8 get configmap demo -n watch-lab --watch --allow-watch-bookmarks. The verbose output shows the HTTP request and confirms that the watch is a long-lived request with resource-version and bookmark parameters. Stop it with Ctrl-C.

Failure modes, security, and debugging

Hot-looping on 410: a controller retries an expired cursor forever. Instrument error reasons and count relists; an expired watch must transition to LIST.
Event-as-command logic: a handler assumes every intermediate update arrives exactly once. Queue a key and reconcile current state instead; events can repeat, coalesce, or arrive after a delete.
Unbounded relist cost: thousands of controllers all relist after an API-server restart. Use shared informers, exponential backoff, jitter, and bounded concurrency.
Selector drift: a watch selector differs from the list selector. The local cache can then contain objects the controller never listed, or miss objects it believes it watches. Treat list and watch configuration as one unit.
RBAC leakage: a watch is a live stream of every matching object the caller may read. Grant least-privilege verbs and resource/namespace scope; do not assume watch is safer than list.
Resource exhaustion: each watch consumes API-server connections, goroutines, serialization work, and client-side memory. Prefer informer sharing and monitor open watches, watch duration, bytes, and relist rate.
Stale authorization assumptions: permissions can change while a stream is open. Handle stream termination and reauthorization errors; never treat an old watch connection as a permanent capability.
Slow consumer: a controller that cannot drain its stream falls behind and eventually needs a relist. Separate watch reading from expensive reconciliation so the stream reader stays responsive.

A useful incident dashboard has four layers: API-server watch count and latency; cache/watch-cache eviction and list cost; client reflector relists and queue depth; and reconciler success, conflict, and API write rate. Looking only at “controller replicas are healthy” misses the failure boundary.

Production design checklist

Use shared informers where multiple reconcilers watch the same kind.
Make the local cache disposable and reconciliation idempotent.
Handle EOF, timeout, authentication errors, and 410 differently.
Use a bounded workqueue with backoff; do not turn API pressure into unlimited retries.
Keep list and watch selectors, namespaces, and API versions identical.
Track logical progress: last event version, last bookmark version, last successful relist, and cache age.
Test API-server restart, etcd compaction, permission changes, duplicate events, rapid delete/recreate, and a slow consumer.
Measure the recovery path. A controller that is correct only while its first watch stays connected is a demo, not a production controller.

Primary sources and implementation landmarks

Kubernetes API concepts — LIST/WATCH, resourceVersion, bookmarks, selectors, and watch behavior.
Kubernetes resource version semantics — consistency choices and why resource versions are opaque.
client-go Reflector source — list/watch lifecycle and relist behavior.
client-go SharedInformer source — shared watch/cache ownership for controllers.
kube-apiserver cacher source — watch-cache fan-out, replay, and cache boundaries.
KEP-1904: watch bookmarks — motivation and protocol design for progress events.
Kubernetes API concepts: resource versions — expired versions and the 410 recovery contract.
etcd API and watch documentation — the storage-level revision/watch model beneath the API server.