chaowen. · Notes

Kubernetes / AI infrastructure / security · 2026-10-11

Kubernetes AI factories: tenant isolation is a hardware boundary

Sharing a GPU cluster is not the same as sharing a web-server cluster. A namespace can separate Kubernetes names and permissions, but it does not automatically isolate packets, accelerator memory, device access, or the management interfaces that configure them. A credible multi-tenant AI platform must define and test each boundary.

Start with the threat model, not the namespace

Suppose two teams run workloads on one cluster. Kubernetes RBAC can prevent team A from reading team B's Pods. NetworkPolicy can restrict Pod traffic—but only when the installed network plugin enforces it. Device plugins expose accelerators to containers, while the device's own partitioning and reset semantics determine what those containers actually share.

Isolation is therefore layered:

Why multi-fabric networking matters

AI clusters often have a high-bandwidth data fabric for collective GPU traffic and a separate ordinary network for control, storage, and services. A workload may need the fast fabric without gaining broad access to another tenant's data-plane endpoints. Isolation must span the path: Kubernetes attachment, virtual network or VLAN/VRF, switch policy, and the host's device configuration. One permissive hop defeats the intended boundary.

In a Kubernetes environment, verify the actual installed CNI and any secondary-network mechanism; a policy object existing in the API is not proof that packets are filtered. Test from a tenant Pod, then inspect fabric counters or flow logs to confirm the denied traffic never reached the other tenant.

A small reproducible policy lab

With a cluster where your CNI enforces Kubernetes NetworkPolicy, create two namespaces and try cross-namespace traffic before and after default-deny. This lab tests the Pod-network layer only—it says nothing about GPU memory or the physical fabric.

kubectl create namespace tenant-a
kubectl create namespace tenant-b
kubectl -n tenant-b run echo --image=nginx --port=80
kubectl -n tenant-a run probe --image=curlimages/curl -- sleep 3600
kubectl -n tenant-a exec probe -- curl -m 2 http://echo.tenant-b.svc.cluster.local

Apply a default-deny ingress policy in tenant-b:

kubectl -n tenant-b apply -f - <<'YAML'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-by-default
spec:
  podSelector: {}
  policyTypes: [Ingress]
YAML
kubectl -n tenant-a exec probe -- curl -m 2 http://echo.tenant-b.svc.cluster.local

The second request should time out or fail to connect when enforcement is active. If it still succeeds, check CNI support, Pod labels, service routing, and policy events before declaring isolation achieved. Clean up with kubectl delete namespace tenant-a tenant-b.

Capacity and isolation can conflict

Whole-GPU assignment is easy to reason about, but can strand capacity. Hardware partitioning can improve utilization, but only if the accelerator supports a defensible partition boundary and the orchestration stack maps it correctly. Time-slicing increases sharing further and may provide scheduling fairness without strong fault or side-channel isolation. Choose based on the tenant threat model, not just utilization charts.

Primary sources

Browse all notes · Home