Kubernetes EndpointSlices: scaling service discovery without one giant object

EndpointSlices solve a deceptively expensive API problem: a Service with thousands of backends should not force every watcher to repeatedly download and process one enormous Endpoints record. Slices shard the backend set into smaller API objects. That reduces update payloads and enables incremental observation, but makes correct consumers responsible for aggregating multiple objects and tolerating asynchronous change.

Problem: one endpoint list becomes a control-plane hotspot

A Service is a stable virtual name; its actual destinations change as Pods become ready, restart, or move. Historically, Kubernetes represented all destinations in one Endpoints object. For a large service, a single Pod change can rewrite a large payload, fan it out to many API watchers, and trigger client-side work proportional to the entire backend set. The pressure is on the API server, network, and every controller or proxy consuming the object—not just the application.

Representation and data flow

EndpointSlices partition endpoints into several objects associated with a Service. Each slice carries address family, protocol, port information, and endpoint records with conditions such as readiness, serving, and termination. The controller manages these objects; consumers watch slices selected for the Service and combine them into their local view.

Pod readiness / membership change
        ↓
EndpointSlice controller
        ↓ create, update, delete slices
Kubernetes API watch stream
        ↓
 kube-proxy / ingress / custom consumer
        ↓ aggregate + apply eligible addresses

The key change is granularity: a consumer may receive an update to one slice rather than a monolithic list. It still needs a coherent logical endpoint set. Watch events are not a globally atomic snapshot across all slices; clients should use the standard list-then-watch pattern, resource versions, and relist after watch expiration or errors.

Conditions are not interchangeable

ready indicates eligibility for ordinary traffic under the relevant Service policy; serving describes whether the endpoint is serving; terminating marks shutdown in progress. Consumers must follow Kubernetes API semantics and account for feature/version behavior rather than treating every address as ready. During graceful shutdown, a terminating endpoint can still be serving briefly while no longer being a preferred new destination. Misreading these fields can cause traffic to hit a process that has begun draining, or prematurely remove a backend that is still completing existing work.

Why sharding helps—and what it costs

Smaller slices reduce the amount of data changed and transmitted for typical membership updates. The slice size is a controller choice, not a permanent application contract. A service may have multiple slices, and consumers must not assume one slice per Service, stable slice names, or a particular endpoint count per slice.

Benefit: smaller watch updates and less repeated serialization for large backend sets.
Cost: more API objects and more bookkeeping; aggregators need set semantics and must handle additions, updates, and deletion.
Tradeoff: smaller shards can reduce per-update payload but increase object churn; oversized shards recreate larger updates.
Compatibility: old clients may still consume the legacy Endpoints API, so migration is not always an instantaneous replacement.

EndpointSlices do not make endpoint propagation synchronous. A Pod can become ready before every consumer has applied the corresponding slice update. Applications still need readiness probes, connection retry behavior, and graceful connection draining.

Security and failure modes

Endpoint addresses are infrastructure metadata. RBAC should grant list/watch access only to principals that need service topology; endpoint data can reveal internal IPs and service structure. A compromised controller or overly permissive writer can redirect traffic by changing slice membership or ports, so writes belong to trusted control-plane components.

Stale cache: a consumer missed watch events or failed to relist after resource-version expiration.
Partial aggregation: code watches only one slice and silently drops backends.
Wrong address family: IPv4 and IPv6 slices are mixed or an unsupported family is selected.
Port mismatch: consumers ignore slice port/protocol data and connect to an incorrect target.
Readiness confusion: terminating or unready endpoints receive new traffic due to simplistic filtering.
Thundering update: a rollout changes many Pods at once, creating a burst of slice events and downstream config pushes.

Reproducible inspection lab

On a disposable Kubernetes cluster, create a Service selecting a few Pods, then inspect slices. This lab only reads API state after workload creation; it does not alter cluster networking.

kubectl create deployment web --image=nginx --replicas=3
kubectl expose deployment web --port=80 --target-port=80
kubectl get service web
kubectl get endpointslices -l kubernetes.io/service-name=web -o yaml

Observe addresses, ports, address type, readiness conditions, and slice labels. Increase replicas and watch changes:

kubectl get endpointslices -l kubernetes.io/service-name=web -w

In another terminal, inspect endpoint count and Pod readiness:

kubectl get pods -l app=web -o wide
kubectl describe endpointslice -l kubernetes.io/service-name=web

Exact output and default slice sizing vary by Kubernetes release and cluster configuration. Verify the cluster version before drawing conclusions from a missing field. Remove only the lab objects when finished: kubectl delete service web deployment web.

Production debugging workflow

Confirm Service selector, namespace, port, and address family.
List every EndpointSlice carrying kubernetes.io/service-name; inspect all slices, not only the first result.
Compare endpoint addresses and conditions with matching Pods, readiness probes, and termination state.
Check the consumer's cache/watch health, last successful relist, resource-version errors, and event queue lag.
Validate the data plane separately: a correct API slice does not prove kube-proxy, ingress, or a custom proxy installed the new configuration.
During incident mitigation, avoid manually editing controller-owned slices unless you understand reconciliation; the controller may overwrite the change or create conflicting state.

Useful metrics include slice count, endpoints per slice, API watch restarts, propagation latency, and backend count seen by the data plane. Correlate control-plane updates with actual connection failures rather than assuming API correctness equals traffic correctness.