Kubernetes / cloud-native control-plane deep dive

Kubernetes leader election: why a Lease is coordination, not fencing

Several replicas may be alive, but only one should run a singleton job: a controller, scheduler, garbage collector, or migration worker. Kubernetes solves the selection problem with a tiny Lease object and optimistic updates. That gives you a leader; it does not magically stop an old leader that is paused, partitioned, or holding an external resource.

Kubernetes / cloud-native control-plane deep dive · Lease objects, resourceVersion compare-and-swap, client-go control flow, failure modes, fencing, debugging, and a local simulation

2026-09-19

The concrete problem

Suppose a deployment has five identical workers but exactly one must renew a certificate, advance a database migration, or publish a periodic snapshot. Running the job in every replica duplicates work. Electing a leader once at startup is not enough: processes crash, nodes pause for minutes, networks partition, and pods are rescheduled.

The system needs a continuously renewed claim with three properties:

• Any replica can observe who currently holds the claim.
• Only one contender should win a contested acquisition in the API server's serialized view.
• A contender that stops renewing should eventually become replaceable.

Kubernetes puts that claim in the coordination API as a Lease. A leader writes its identity and renewal timestamp. Followers read the object and attempt an update only when the current record is expired or already theirs. The API server's resourceVersion turns the update into a compare-and-swap.

What a Lease contains

The coordination.k8s.io/v1 Lease API deliberately stores very little. The important fields are:

Modern Kubernetes also uses Leases for node heartbeats and, in some components, API-server identity. Do not infer that every Lease is a leader-election lock: inspect the namespace, name, annotations, and owning component. The data structure is shared; the protocol built around it is the real contract.

The protocol: read, decide, compare-and-swap

Let D be the configured lease duration and R the last observed renewTime. A contender treats the record as valid while:

now < R + D

If the record is held by another identity and is still valid, the contender waits. If the record is expired, it copies the observed record, changes holderIdentity to itself, sets a new renewTime, and sends an update containing the observed resourceVersion.

The API server is the serialization point. The loser is not determined by who read first or who has the fastest local clock; it is determined by which valid update reaches the storage path first. A 409 Conflict is normal contention, not necessarily an incident.

client-go control flow

The reference implementation used by many Kubernetes controllers is in client-go/tools/leaderelection. Its public shape is intentionally callback-based:

leaderelection.NewLeaderElector(leaderelection.LeaderElectionConfig{
    Lock:            resourceLock,
    LeaseDuration:   15 * time.Second,
    RenewDeadline:   10 * time.Second,
    RetryPeriod:      2 * time.Second,
    Callbacks: leaderelection.LeaderCallbacks{
        OnStartedLeading: runSingletonWork,
        OnStoppedLeading: stopWork,
        OnNewLeader:      observeLeader,
    },
    ReleaseOnCancel: true,
})

Names vary slightly with the client-go release, but the state machine is stable:

1. Run(ctx) calls an acquisition loop until the process creates or successfully updates the lock.
2. After acquisition, OnStartedLeading runs the singleton workload, usually in a goroutine.
3. A renewal loop periodically reads and updates the record. It must complete within RenewDeadline, not merely before the longer LeaseDuration.
4. If renewal fails for the deadline, the elector cancels the leading context and invokes OnStoppedLeading. A well-behaved worker exits or stops all singleton side effects.
5. Other replicas observe the changed or expired record, win an update, and receive OnStartedLeading.

In the source, the important concepts are LeaderElector.acquire, LeaderElector.renew, tryAcquireOrRenew, the ResourceLock interface, and LeaderElectionRecord. The lock abstraction separates the protocol from the storage object; the standard implementation uses the coordination API's Lease resource. The code is worth reading because the timing rules are not equivalent to “renew every N seconds”: it observes API errors, retries within a deadline, and tracks local timing to decide when to give up.

The timing triangle

Three settings are easy to misconfigure:

As a practical invariant, the deadline must be meaningfully shorter than the lease duration, and the retry period must be small enough to obtain several attempts inside the deadline. The exact ratio depends on API-server tail latency, client throttling, node scheduling, and the number of contenders. Measure p99 and p999 behavior; do not tune from the happy-path average.

Clock behavior matters. The protocol relies on local time observations, so an implementation needs a monotonic clock for elapsed-time comparisons. Wall-clock jumps, VM suspension, long garbage-collection pauses, CPU starvation, and overloaded API servers can all make a leader miss its renewal window. A short lease that looks elegant in a test can produce election storms in a busy cluster.

Why this is not fencing

Consider a stop-the-world pause or a network partition:

The client-go package explicitly warns that it does not guarantee that only one client is acting as leader, a property commonly called fencing. Kubernetes can cancel a context and ask A to stop, but it cannot forcibly interrupt every system call, revoke a TCP connection at a third-party service, or undo a write already accepted by an external system.

For idempotent cache refreshes, duplicate metrics, or a best-effort controller, this may be acceptable. For money movement, schema migration, volume ownership, or a non-idempotent external API, add a real fencing mechanism:

The pattern is: leader election gives a process a lease epoch; the downstream resource must validate that epoch before accepting a side effect. Without the second half, the Lease is advisory coordination.

Reproducible lab: make the compare-and-swap visible

Goal: observe one contender winning an optimistic update and then observe why a stale process can still perform a side effect. This lab uses only Python's standard library and models the API-server serialization point; it does not contact a cluster.

python3 - <<'PY'
record = {"holder": "A", "resourceVersion": 41, "expires": 10}

# Both contenders read the same expired record before either update commits.
a_read = record.copy()
b_read = record.copy()

def cas(expected_rv, candidate):
    global record
    if record["resourceVersion"] != expected_rv:
        return False
    record = candidate
    return True

b_candidate = {"holder": "B", "resourceVersion": 42, "expires": 25}
a_candidate = {"holder": "A", "resourceVersion": 42, "expires": 25}

print("B update:", cas(b_read["resourceVersion"], b_candidate))
print("A update:", cas(a_read["resourceVersion"], a_candidate))
print("stored record:", record)

# Election changed, but it cannot revoke an already-running old process.
effects = ["A write before pause"]
effects.append("B write after taking over")
effects.append("A stale write after resuming")
print("possible external effects:", effects)
PY

Expected result: B succeeds, A fails its stale compare-and-swap, and the stored holder is B. The final list nevertheless contains an A write after takeover. That last line is the point: the record is linearizable coordination, not a kill switch.

Variation 1 — add fencing: attach epoch=42 to B's downstream write and make the receiver store the greatest accepted epoch. A's later write carries epoch=41 and is rejected. This is the missing safety property in systems that cannot tolerate stale leaders.

Variation 2 — inspect a real Lease: if a Kubernetes context is available, run:

kubectl create namespace leader-lab
kubectl -n leader-lab apply -f - <<'YAML'
apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: demo
spec:
  holderIdentity: worker-a
  leaseDurationSeconds: 15
  acquireTime: "2026-09-19T00:00:00Z"
  renewTime: "2026-09-19T00:00:00Z"
  leaseTransitions: 1
YAML
kubectl -n leader-lab get lease demo -o yaml
kubectl -n leader-lab delete namespace leader-lab

The API server adds metadata.resourceVersion. Repeat the get while a real controller renews the object and watch renewTime and the version change. On this machine, kubectl is installed but its configured server returned NotFound for /version, so the live-cluster variation could not be run here; the standard-library simulation is the verified fallback.

Failure modes and debugging

A good operational dashboard separates four signals: current Lease holder, time since last renewal, transition rate, and singleton-work success/error rate. A pod being Ready says almost nothing about leadership. Conversely, a Lease with a fresh timestamp does not prove that the holder is healthy outside the API server.

For incident work, save the Lease YAML, pod UIDs, controller logs, API-server audit events, and node scheduling/pressure data. Correlate renewTime with request latency and process pauses. If an external write was duplicated, investigate receiver logs and credential scope; deleting the old pod after the fact is cleanup, not proof that the old write could not happen.

Security and design implications

The Lease is a control-plane authorization signal, so its permissions matter. A controller needs the minimum get, create, and update permissions on the particular Lease or namespace it owns. Broad write access lets a compromised workload seize another component's lock, induce failover churn, or prevent a singleton from running. Use a dedicated service account, a narrow Role, and a stable, non-user-controlled Lease name.

Do not treat the holderIdentity string as authentication. It is metadata supplied by the client. The API server authenticates the service account making the update; the application must choose an identity scheme that prevents accidental collisions. If the selected leader also handles sensitive work, keep its workload permissions separate from the permission to update the coordination object where practical.

Finally, avoid turning a Lease into a home-grown distributed transaction. The API server gives you a useful linearizable point for the record, not atomicity across Kubernetes and an external database. Either make the work idempotent or carry a fencing epoch into the system that owns the side effect.

Production checklist

• Use a unique per-process identity, commonly including pod name and UID.
• Choose duration, deadline, and retry values from measured tail latency and pause behavior.
• Propagate the leading context to every worker and wait for shutdown.
• Assume a short overlap is possible; make the workload idempotent or add receiver-side fencing.
• Alert on transition churn, stale renewals, and duplicate downstream effects—not only on pod restarts.
• Scope RBAC to the exact Lease and service account; audit who can update it.

The useful mental model is not “Kubernetes guarantees one leader.” It is: “Kubernetes offers a renewable, compare-and-swap claim; my application must stop on loss, and the system receiving irreversible side effects must reject stale claims.” That distinction is small in documentation and enormous in production.

Sources and implementation references

• Kubernetes documentation: Leader election — the supported concept and operational model.
• Kubernetes API reference: Lease v1 — fields and API contract.
• client-go leaderelection.go — acquisition, renewal, deadlines, callbacks, and the explicit no-fencing warning.
• client-go LeaseLock — mapping between the leader record and the Lease API.
• Kubernetes coordination API types — open-source definition of Lease fields.
• The Chubby lock service for loosely-coupled distributed systems — the classic lock-service discussion of leases, failure, and stale clients.