Cloud platforms / failure-domain engineering deep dive
2026-10-05
Cloud sovereignty under disconnection: proving a failure domain is truly local
A cloud can keep customer data in one jurisdiction and still depend on remote systems to operate. The hard engineering question is whether the environment can keep serving, managing, securing, and recovering workloads after selected external dependencies disappear. A planned network disconnection is a useful test because it turns an architectural claim into an observable failure experiment.
1. Residency is not independence
Data residency is a statement about where specified data is stored or processed. Operational independence is a stronger, different property: a region can continue to provide agreed services when some outside systems, people, networks, or software supply paths are unavailable. A database may store every row locally yet fail to authenticate administrators because identity depends on a foreign control plane. A virtual machine may keep running while operators cannot renew certificates, publish security rules, or replace a failed host.
This distinction matters in regulated environments, disaster recovery, sanctions planning, and ordinary resilience engineering. “Sovereign” is not a protocol guarantee. It is a claim over a defined service scope, threat model, jurisdiction, and set of dependencies. Without those boundaries, a successful outage test can be technically true but operationally meaningless.
2. Model the dependency graph
Represent the service as a directed graph. Nodes are components or capabilities; an edge A → B means A requires B for a specified operation. Include more than runtime calls: DNS, identity, time, key management, artifact delivery, telemetry, support access, network routing, and people who hold an approval role all create dependencies.
Annotate every edge with criticality, direction, data classification, failure behavior, and freshness requirements. “Cached” is not equivalent to independent: a cached credential expires; a cached policy may be stale; an image mirror may lack the patched artifact needed for recovery. Distinguish the ability to continue existing work from the ability to perform lifecycle operations such as scaling, deploying, rotating keys, or restoring data.
Define the cut set
A disconnection test removes a set of edges, called a cut set, rather than vaguely “turning off the internet.” For example, remove the private backbone path while retaining approved local ISP connectivity; separately test loss of a global identity endpoint or artifact mirror. Record what remains reachable. If the claim is independence from non-local infrastructure, enumerate every path that crosses the boundary—including backup routes, remote observability exports, and management callbacks.
For a service graph, a single mandatory external node is a single point of dependency. Redundancy inside one jurisdiction does not fix an external cut vertex. Conversely, intentionally retaining an encrypted global network path for performance does not by itself prove that customer content leaves the region; data-flow classification and network topology answer different questions.
3. Separate data plane, control plane, and operations
The data plane handles customer requests and stored content. The control plane creates and changes resources: scheduling, IAM evaluation, API front doors, quotas, and configuration distribution. The operations plane gives humans and automation the authority and evidence to observe, repair, and secure the service. These planes have different survival conditions.
A workload may remain healthy in the data plane while the control plane is unavailable. That can be an acceptable bounded degradation if the system has enough local capacity and the outage window is finite. It is dangerous if operators cannot revoke a compromised key or isolate an attack. A truly useful exercise therefore tests both customer requests and safety-critical operator actions: stop a workload, rotate a credential, roll back a deployment, recover a node, and verify an alert reaches the local responder.
Local copies of source code and software are necessary but not sufficient. Rebuilding may require signing keys, trusted time, package metadata, license services, build workers, or a human authorization chain. Locality is a property of the entire recovery path, not just the repository mirror. A break-glass procedure that works only when the external identity provider is online is not a local recovery path.
Graceful degradation should be designed
For each dependency, specify fail-open or fail-closed behavior. Authentication normally fails closed to protect resources, but a short-lived, tightly scoped emergency credential may enable recovery without making the whole identity system permissive. Configuration updates might queue locally and reconcile later. Telemetry may buffer with bounded disk usage. Each fallback needs expiry, audit, replay protection, and a defined return-to-normal procedure.
4. Network withdrawal and convergence
Removing a preferred backbone route does not make traffic instantly appear on another path. Routers detect failure or receive a planned withdrawal, recalculate best paths, and propagate routing updates. During convergence, different networks can hold different views of reachability. Existing transport connections may break; packets can be lost or briefly follow asymmetric paths. A route that exists in a routing table may still fail due to ACLs, MTU, stateful firewalls, or return-path policy.
Planned cutover and abrupt failure test different mechanisms. A graceful withdrawal tests route selection, policy, and expected convergence. A physical/link failure tests detection timers and failure detection. A control-plane partition can leave stale routes advertised. Exercise each separately and measure packet loss, connection resets, DNS resolution, application error rate, and time to stable reachability from multiple external vantage points.
Do not interpret “service remains available” as “no user-visible impact.” State the SLO and measurement window. A service can meet a monthly availability objective while exhibiting several seconds of failover disruption; that may be unacceptable for a payment authorization but harmless for an asynchronous batch job. Keep private customer connectivity distinct from public egress: changing one path need not imply the other changed.
5. Reproducible lab: discover a hidden mandatory edge
Goal: model whether local service operations survive loss of one dependency. Prerequisites: Python 3.8+; run locally or in a disposable environment. This is a small graph exercise, not a cloud emulator and not proof of any provider’s architecture.
from collections import defaultdict, deque
edges = {
"serve": ["regional-api", "local-db"],
"regional-api": ["local-auth", "global-policy"],
"recover": ["regional-api", "local-image-mirror", "local-signing"],
}
def reachable(start, unavailable):
seen, queue = set(), deque([start])
while queue:
node = queue.popleft()
if node == unavailable or node in seen:
continue
seen.add(node)
queue.extend(edges.get(node, []))
return seen
for operation in ("serve", "recover"):
for outage in ("global-policy", "global-backbone"):
print(operation, "without", outage, "=>", sorted(reachable(operation, outage)))Expected observation: removing global-backbone has no effect in this abstract graph because it is not an edge; that is a modeling warning, not a pass. Removing global-policy makes the node unreachable for the service path as written. Add explicit edges for DNS, network, identity, artifact, and telemetry paths, then identify which operations fail. For a real test, replace graph reachability with controlled fault injection and end-to-end probes.
Variation 1: add local-policy-cache as a fallback edge and encode a maximum cache age. Ask what happens when the outage exceeds that age; a cache without expiry semantics can conceal a dependency.
Variation 2: test a failure on recovery only: keep serving traffic, but remove the image signer or deployment API. This reveals the “running but unrecoverable” state, often missed by availability-only checks. Never run disruptive tests against production without an approved scope, rollback path, and customer-impact controls.
6. Evidence, failure modes, and security
A defensible report records the test boundary, dependencies removed, workloads and operator workflows exercised, start/end timestamps, measured impact, and any exclusions. Preserve route changes, service health, access logs, alert delivery, and audit events. Evidence must show both that the intended external edges were cut and that no hidden route remained. A result from one exercise proves behavior under that tested configuration and interval; it does not prove every service or every future release independent.
- False independence from warm caches: the exercise is shorter than token, certificate, DNS, or policy expiry. Repeat beyond relevant TTLs or demonstrate local renewal.
- Partial service coverage: a web request succeeds while restore, scale-out, or credential revocation fails. Define operational workflows as explicit acceptance tests.
- Silent bypass: fallback routing works, but crosses a prohibited jurisdiction or exports operational metadata. Inspect flows and classify data; encryption protects confidentiality in transit, not location or metadata policy by itself.
- Local control-plane compromise: isolation can reduce external dependencies while concentrating privileged authority in a smaller operator group. Apply separation of duties, hardware-backed keys, immutable audit, scoped break-glass access, and independent review.
- Recovery creates split brain: a partitioned site may continue accepting writes while another control plane believes it owns the resource. Use fencing epochs, quorum rules, or explicit single-writer authority; connectivity restoration must reconcile state safely.
- Test artifact is not reproducible: manual route changes or undocumented exemptions make results hard to trust. Version the test plan, configuration, probes, and evidence collection; repeat after material architecture changes.
Debug by correlating the dependency graph with traces and control-plane audit logs. Tag spans with region, dependency, and fallback mode; alert on unexpected cross-boundary calls. During the exercise, compare planned and observed flows, inspect DNS answers and route tables at multiple points, and probe actual user journeys rather than only ICMP. Afterward, verify that routes, policy, and emergency credentials returned to their intended state. The rollback itself is part of the test.
Sources
- AWS: European Sovereign Cloud—demonstrating an independent operation — planned October 24, 2026 exercise, stated scope, network cutover, and caveats. This is a provider description of a planned test, not results of the future exercise.
- Cloud Computing News: AWS to disconnect European Sovereign Cloud from global network — independent contemporaneous reporting on the announcement.
- RFC 4271: A Border Gateway Protocol 4 (BGP-4) — route advertisement and withdrawal protocol foundations.
- RFC 7938: Use of BGP for Routing in Large-Scale Data Centers — operational routing design and convergence context.
- NIST SP 800-34 Rev. 1: Contingency Planning Guide for Federal Information Systems — business impact, contingency planning, and recovery testing.