Kubernetes control loops / API design

Kubernetes controllers: who owns what from watch to reconcile

An informer is not a controller, a watch event is not a job, and an Operator is not a magical deployment format. Kubernetes works because each layer has a narrow contract: the API stores intent, list/watch distributes change, informers maintain a local view, queues schedule keys, and reconcilers make reality converge. Labels, annotations, spec, and status exist to keep those contracts from collapsing into one convenient but ungovernable map.

Kubernetes deep dive · Controllers, watches, and API contracts

2026-08-18

The whole machine in one picture

The arrows form a loop, not a pipeline that handles each event exactly once. A controller can crash after performing a side effect but before updating status. A watch can disconnect. Several updates can collapse into one queued key. A resync can enqueue an object that did not change. Correctness therefore cannot depend on receiving a perfect event history.

Responsibility map

Subscription means LIST, then WATCH

A fresh controller cannot safely start with “tell me what changes next.” It first needs the objects that already exist. Kubernetes returns a collection plus a collection resourceVersion; the client then watches from that version. The resulting stream contains ADDED, MODIFIED, DELETED, and sometimes BOOKMARK or ERROR events.

resourceVersion is an opaque concurrency and continuation token, not a timestamp or portable sequence number. Watch history is finite. If the requested version has been compacted, the server can return 410 Gone; the client must rebuild its snapshot with another list and resume. A standard Reflector exists so every controller author does not reimplement this state machine badly.

Watch is also not Kafka. It does not promise durable replay forever, consumer offsets, or exactly-once work. Its purpose is efficient cache synchronization. The durable thing is the current object in the API, so recovery returns to current state rather than reconstructing business truth from old notifications.

Where filtering belongs

“Filter the subscription” hides three different decisions. They have different cost and correctness properties.

Use server-side labels when they define a stable population: for example, one controller shard managing control-plane.example.com/shard=west. Use client-side predicates when the condition is richer, changes often, or depends on old-versus-new state. Field selectors are intentionally limited to fields each resource exposes; they are not arbitrary JSONPath queries. Annotations have no native selector API.

Informer anatomy: transport, cache, notification

“Informer” is commonly used for the whole client-side mechanism, but its pieces deserve separate names:

• Reflector: performs list/watch, tracks the last resource version, applies backoff, and replaces local state after a relist.
• DeltaFIFO: buffers object deltas between the Reflector and informer processing. It is an implementation queue for synchronizing the cache, not the controller’s retry queue.
• Store / Indexer: holds the latest object snapshots and optional secondary indexes such as namespace or owner UID. Listers read here without calling the API server.
• Shared processor: fans add, update, and delete notifications out to registered handlers so many controllers can share one watch and cache.
• Resource event handler: translates a potentially interesting object notification into a key for a controller workqueue.

The controller workqueue is deliberately separate from DeltaFIFO. Informer internals care about synchronizing object snapshots. The workqueue cares about retrying reconciliation. Queue namespace/name, not a pointer to the event object: by the time a worker runs, that object may be obsolete, while the cache contains a newer version.

Wait for cache sync before starting workers. Otherwise an empty cache is indistinguishable from “the cluster contains no matching object,” which can trigger destructive cleanup. Never mutate an object returned by a lister; informer objects are shared. Deep-copy before modification, then write through a typed or dynamic client.

A periodic informer resync does not necessarily poll the API server or invent a missing watch history. It re-delivers current cached objects to handlers so level-driven logic gets another chance. Treat it as a safety nudge, not the primary retry mechanism. Reconcile errors belong in the rate-limiting workqueue with explicit backoff.

Controller versus Operator

A controller is a control loop. An Operator is a product and design pattern: an application-specific controller, usually paired with custom resources, that encodes knowledge a skilled human operator would otherwise apply. The term was introduced by CoreOS in 2016 while explaining how to automate stateful systems such as etcd and Prometheus—upgrades, membership changes, backup, and recovery—not merely create a Deployment.

Good Operators reuse built-in primitives. A database Operator may own a StatefulSet, Services, Secrets, Jobs, and PVCs; it should not rebuild scheduling or storage attachment. Its special value is the ordering and policy between those resources and the external application state.

One object, four different kinds of information

apiVersion: database.example.com/v1
kind: Database
metadata:
  name: orders
  labels:
    app.kubernetes.io/name: orders-db
    platform.example.com/tier: critical
  annotations:
    platform.example.com/runbook: https://runbooks.example.com/orders
    platform.example.com/change-ticket: CHG-2048
spec:
  engine: postgres
  version: "18"
  replicas: 3
  backup:
    schedule: "0 3 * * *"
status:
  observedGeneration: 7
  readyReplicas: 3
  currentVersion: "18"
  conditions:
  - type: Ready
    status: "True"
    reason: ClusterHealthy

metadata.generation usually advances when desired state changes. A controller copies the generation it has processed into status.observedGeneration. Without that handshake, Ready=True might describe the previous spec. Conditions should answer operational questions with type, status, reason, message, and transition time; they should not become an append-only event log.

Why not put everything in labels?

Because convenience at write time becomes coupling everywhere else. Labels are Kubernetes’ core grouping primitive. Services, workload controllers, affinity rules, policies, CLIs, UIs, and LIST/WATCH clients use selectors to turn labels into populations. Changing a label can reroute traffic, transfer controller attention, change scheduling, or alter policy membership. That is a control surface, not a junk drawer.

The split is historical and architectural. Kubernetes needed cross-cutting sets that did not force objects into one rigid hierarchy: “production,” “frontend,” “canary,” and “customer A” can overlap. Labels provided that loose, selector-friendly identity. It also needed metadata that tools could attach without making it part of grouping semantics or forcing an external database. Annotations became that non-identifying escape hatch. Typed API fields and the spec/status convention then provide evolvable contracts where metadata maps are too weak.

Annotations are not harmless merely because they are unselectable. Controllers often use them as opt-in directives, and Kubernetes itself has well-known annotation contracts. Use a DNS-qualified key, document ownership, and avoid multiple writers. Promote a widely used, essential annotation into a typed spec field when you control the API. Annotations are useful for extension and compatibility; they are poor permanent homes for a product’s central configuration.

Ownership, children, and deletion

A controller should make ownership explicit. ownerReferences express lifecycle and garbage-collection relationships; labels help find and group children but do not establish ownership. Before adopting or deleting a child, compare owner UID, not only name or labels: names can be reused after deletion.

Finalizers are another contract boundary. Setting a finalizer says, “do not complete deletion until this controller performs required cleanup.” Once deletionTimestamp appears, reconciliation should enter a dedicated finalize path, make cleanup idempotent, and remove only its own finalizer. A finalizer is not a guarantee that an unreachable external service can be cleaned forever; define timeouts, abandonment policy, and observability so an object does not remain terminating invisibly.

For writes, prefer narrow patches and clear field ownership. The spec writer should not overwrite status. The status writer should not rewrite spec. Two controllers should not race over the same condition or annotation key. Server-Side Apply can track field managers, but it cannot repair a confused ownership model.

A reconcile shape that survives reality

func Reconcile(key NamespacedName) error {
    obj := cache.Get(key)
    if obj == nil {
        // Object is gone. Usually nothing to do unless external cleanup
        // is represented elsewhere; finalization should happen before deletion.
        return nil
    }

    if obj.DeletionTimestamp != nil {
        return finalizeIdempotently(obj)
    }

    desired := renderChildren(obj.Spec)
    actual := readCachedChildren(obj.UID)

    // Patch only owned fields; repeated calls converge.
    apply(desired, actual)

    observed := inspectApplicationState(obj)
    patchStatusIfChanged(obj, observed, obj.Generation)
    return nil
}

Real code needs timeouts, context cancellation, conflict retries, per-operation error classification, and rate limits. Still, this skeleton has the important shape: fetch current state by key, branch on deletion, compute desired state from spec, observe owned reality, make bounded changes, and publish status only when it changed.

Avoid doing an entire upgrade in one reconciliation. Encode phase in real observed resources or typed status, perform one safe step, and requeue. Long synchronous workflows hold workers, complicate leader changes, and make crash recovery dependent on call stacks instead of durable state.

Common boundary failures

• Event-driven business logic: “OnAdd creates X; OnDelete removes Y.” Recovery breaks when events coalesce or the process restarts. Handlers should enqueue; reconcile should derive both actions from current state.
• API call inside informer callback: slow I/O blocks notification fan-out and grows lag. Put the key on a workqueue.
• Reading your write from cache immediately: the watch may not have delivered it yet. Design eventual convergence, or use a direct uncached read only when the operation truly needs fresh confirmation.
• Using annotations as an unversioned spec: parsing JSON strings moves schema, defaults, and errors into controller runtime. Users discover mistakes only after admission succeeded.
• Using labels as ownership: another object can share or change the label. Verify owner references and UIDs before destructive action.
• Status write loop: patching an identical status creates another update, which triggers another reconcile. Compare before writing and predicate on meaningful changes, often generation.
• Assuming resync means fresh server state: resync can replay cached objects. Use it as a reconcile nudge, not proof of a new LIST.
• One giant Operator: unrelated domains share queues, permissions, failure modes, and release cadence. Split controllers by clear ownership even if one binary hosts several loops.
• Filtering only in the handler: saves reconciles but not API-server traffic or cache memory. Move stable population filters into LIST/WATCH options when scale requires it.

The historical logic, compressed

1. Resources and controllers: Kubernetes modeled intent as API objects and used many small control loops rather than one orchestration transaction. Failure recovery becomes “re-read and converge.”
2. Labels: flat, overlapping, user-defined dimensions let clients select sets without forcing infrastructure-owned hierarchy. They became the API’s grouping and routing primitive.
3. Annotations: tools still needed arbitrary non-identifying metadata near the object. Keeping it out of selectors preserved freedom in size and characters and reduced accidental semantic coupling.
4. Spec and status: separating desired input from observed output lets users and controllers write different subresources, supports generation tracking, and prevents observation from masquerading as intent.
5. LIST/WATCH and informers: many controllers polling full collections would crush the API server. Shared client caches turn one synchronized stream into cheap local reads for many handlers.
6. Operators: once core primitives handled generic scheduling and lifecycle, CoreOS’ 2016 pattern encoded the missing application runbook for stateful systems in custom resources plus controllers.

The boundaries are not ceremony. Each arose because a single mechanism stopped scaling semantically or operationally. “Put it all in labels” feels elegant only while one author, one controller, and ten objects exist.

Design checklist

• Can reconciliation recover from a restart using only API and external current state?
• Do event handlers only enqueue stable keys?
• Are caches synced before workers can interpret absence?
• Does each server-side selector define a stable population the controller can safely ignore outside?
• Are labels limited to useful grouping dimensions with controlled cardinality?
• Is essential behavior represented by typed spec instead of annotation strings?
• Does status report observation, include observedGeneration, and avoid no-op writes?
• Are child ownership and deletion based on owner UID and finalizers, not label coincidence?
• Can every side effect be retried without corrupting state?
• Does the Operator encode real domain operations while reusing built-in Kubernetes primitives?

Primary sources

• Kubernetes API concepts — list/watch, resource versions, bookmarks, compaction, and 410 Gone recovery.
• client-go tools/cache — Reflector, stores, indexes, DeltaFIFO, handlers, and cache synchronization.
• Kubernetes controllers — desired/current state and the many-small-control-loops design.
• Labels and selectors — identifying attributes, grouping motivation, syntax, and LIST/WATCH selection.
• Annotations — arbitrary non-identifying metadata, syntax, and size constraints.
• Kubernetes API conventions — metadata, spec/status, generation, conditions, and API evolution.
• Kubernetes Operator pattern — custom resources, controllers, and application operations knowledge.
• CoreOS: Introducing Operators (2016) — the original motivation and historical examples.