Hardware / CPU security deep dive

Remote Spectre in a serverless isolate: why frozen timers were not enough

A multi-tenant runtime can hide its clock, forbid threads, and still leak a neighboring tenant's secret through speculative execution. The important lesson is architectural: a side channel can cross a language-level boundary even when ordinary loads and stores cannot.

Hardware / CPU security deep dive · transient execution, cache timing, V8 Sandbox, memory-protection keys, and multi-tenant edge runtimes

2026-08-24

The concrete problem

Imagine a serverless provider running thousands of customers' JavaScript programs inside one operating-system process. Giving every request a fresh process or virtual machine would make isolation easier to reason about, but startup latency, memory overhead, and context switching would make short-lived edge functions much more expensive. A JavaScript engine therefore gives each customer a language-level isolate: separate heaps and APIs, shared process and CPU.

Under normal execution, an attacker cannot ask the engine for a pointer to another isolate's heap. Spectre changes the question. The CPU may execute instructions transiently before a branch condition is resolved. Those instructions are later discarded architecturally, but their cache effects remain. If the attacker can encode a secret bit as “cache line A touched” versus “cache line B touched,” timing the next access reveals the bit without an ordinary cross-tenant read.

Why this became news in August 2026

Cloudflare and the University of Edinburgh published Remote-Timer-as-a-Service: Efficient Microarchitectural Leakage in the Cloud with Remote Timers. The researchers evaluated Cloudflare Workers in work performed during 2024 and early 2025, with the paper submitted on August 17, 2026. Cloudflare's engineering account says the demonstrated attack was already mitigated and that they found no indicators of active exploitation.

The result matters because it defeated a plausible defense model, not because it introduced a new JavaScript feature. The production proof of concept used a co-located victim and attacker, cache-signal amplification, and a remote timer to leak a JWT at up to 12 bits per second with 99.16% accuracy in the paper's abstract. That is fast enough to change “theoretically possible” into a serious multi-tenant risk.

The attack as a data-flow problem

A Spectre leak has two separate channels: a transient data path and a timing observation path. Keeping them separate makes the design easier to debug.

1. Train. Call a gadget repeatedly with an object whose type check is true. The branch predictor learns the common path.
2. Confuse. Present an object with an attacker-controlled layout. While the CPU still predicts the old path, transient execution follows a field that should not be trusted.
3. Encode. Use one secret bit to choose one of two distant cache lines. The value itself need not be returned to JavaScript.
4. Measure. Probe both lines. A cache hit is faster than a miss, but the difference is only nanoseconds; a remote network timer is microseconds to milliseconds and noisy.
5. Amplify and repeat. Cache-replacement behavior can turn one hit or miss into many timing events. Repeated samples and a statistical discriminator recover a bit with an error rate low enough to matter.

The Cloudflare paper's production work adds two details that timer-hardening proposals often miss. A WebSocket can provide a long-lived remote measurement path, and Durable Objects can help achieve physical co-location. The attacker does not need a high-resolution clock exposed directly inside the JavaScript API if the system provides a noisy but repeatable external observation point.

What the CPU is actually doing

Out-of-order CPUs predict branches so they can keep their pipelines full. A branch such as if (obj instanceof ObjP) depends on information that may arrive later than the dependent load. The core predicts the branch, begins executing the dependent instructions, and eventually discovers whether the prediction was correct. On a miss, register results and other architecturally visible work are squashed.

Cache state is different. The CPU is allowed to retain fetched lines and replacement metadata because caches are performance structures, not architectural registers. Spectre turns that “harmless” leftover state into a one-bit communication channel:

secret_bit = (transient_value >> bit) & 1;
touch(probeArray + secret_bit * 0x800);

// later, outside the transient path
for (int candidate = 0; candidate < 2; candidate++)
    timing[candidate] = measure(probeArray + candidate * 0x800);

The stride is chosen so the candidate addresses do not occupy the same cache line. Real attacks must also control alignment, eviction, branch-predictor state, JIT-generated code, interrupts, context switches, and cache contention. That is why “I ran a toy Spectre loop once” is not evidence that a cloud defense works—or fails.

Why the first defenses were reasonable, and why they failed

V8 Sandbox: constrain the pointer after heap corruption

V8's software Sandbox addresses a related but distinct failure: a memory-safety or logic bug corrupts an object in the V8 heap, then an attacker uses a corrupted pointer or size to reach the rest of the host process. The design makes the heap a constrained address region and removes direct pointers from sandboxed data wherever possible.

Conceptually, a normal field such as a typed-array backing pointer becomes either a bounded offset or a handle into a protected table:

// conceptual shapes, simplified from V8's sandbox design
struct SandboxedArrayBuffer {
  sandbox_ptr_t  buffer;  // offset from the sandbox base
  sandbox_size_t size;    // bounded sandbox-compatible size
};

struct ExternalArrayBuffer {
  external_ptr_t buffer;  // index/tag into an external pointer table
};

A corrupted 32- or 40-bit offset can still corrupt the sandboxed heap, but it cannot directly name an arbitrary 64-bit address outside that region. An external pointer is resolved through a table whose entry carries a pointer and a type tag; a mismatched tag makes the handle invalid. This is a containment boundary for memory corruption. It does not, by itself, make branch prediction or caches private.

Intel MPK: a hardware tripwire between tenant heaps

Memory Protection Keys (MPK, exposed on Linux through pkey_mprotect() and pkey_set()) attach a small key number to pages. A thread's protection-key register then decides whether that thread may read or write pages carrying each key. Changing the register is cheaper than changing page tables and invalidating TLB entries.

Cloudflare describes assigning a key to each isolate's heap so an accidental or exploit-driven access normally hits a hardware fault. On x86, key 0 is the default and up to 15 additional keys are available per process, so this is not an unlimited one-key-per-tenant scheme. Cloudflare's defense combines MPK with other controls and uses randomization plus a common-case strategy to improve coverage.

pages for isolate A  ── pkey 3 ──┐
pages for isolate B  ── pkey 7 ──┤  page tables retain ordinary R/W bits
thread PKRU: key 3 = allow     │  key 7 = deny  ──► access raises SIGSEGV

// the important caveat
WRPKRU is an unprivileged instruction: arbitrary native code that can
execute it, or a bug that lets an attacker control the register, defeats
this as a standalone security boundary.

MPK is therefore best understood as a fast, hardware-enforced tripwire that turns many wrong-heap accesses into a crash or signal. It is not a replacement for process isolation, pointer discipline, or Spectre-specific mitigations. The Linux manual explicitly warns that pkeys were not primarily designed as a security feature and that applications must manage key lifetime and thread inheritance carefully.

Failure modes in production

• “Timer hidden” becomes “timer moved.” Any externally observable latency can be a measurement surface. Treat I/O, WebSockets, request batching, Durable Objects, and co-located services as part of the threat model.
• Detection races the leak. A detector that needs many invocations or a 30-second window can lose if a signal is amplified and extracted in one invocation. Measure time-to-detection against time-to-exfiltration, not just classifier accuracy.
• MPK key exhaustion and reuse. There are few keys. Reuse, thread inheritance, signal handlers, stale mappings, or incorrect cleanup can turn an isolation optimization into a confused-deputy or availability bug.
• V8 Sandbox coverage gaps. Every pointer-sized field, size, table entry, trusted object, and host callback is part of the boundary. One unmediated escape hatch can restore an arbitrary read/write primitive.
• False confidence from a local benchmark. Cache behavior depends on CPU model, scheduler, JIT version, system load, and topology. A mitigation tested on one laptop is not a characterization of a globally distributed edge fleet.
• Availability impact. A defense that kills a tenant on a pkey fault or isolates it into a process may be secure but can create retries, cold-start spikes, and noisy-neighbor effects. Feed security faults into rate limits and incident telemetry without exposing secrets.

How to debug and evaluate the boundary

1. Draw the boundary at three layers: language objects, virtual-memory pages, and shared CPU structures. Write down which layer each defense actually controls.
2. For a suspected V8 escape, trace the value from a JavaScript object field through its generated machine code, pointer representation, external-pointer-table lookup, and host callback. Inspect whether the value is a sandbox offset, a tagged handle, or a raw host pointer.
3. For a suspected Spectre path, compare architectural output with microarchitectural observations. Use controlled, non-secret data and record cache misses, branch mispredicts, iTLB activity, context switches, core migrations, and timer jitter.
4. Test the detector's worst case: one invocation, high WebSocket/I/O activity, noisy neighbors, JIT warm-up, and frequent runtime upgrades. The question is whether extraction finishes before isolation, not whether an average script looks suspicious.
5. Validate hardware assumptions on every machine class. Check CPU support, kernel support, pkey allocation failure, scheduler behavior, and the fallback path when MPK is unavailable.

Compact reproducible lab: MPK as a page-level tripwire

Goal: make one page readable, assign it a protection key, disable that key for the current thread, and observe a hardware-enforced fault. This demonstrates the hardware layer used for defense in depth; it is intentionally not a cross-tenant Spectre exploit.

Prerequisites: disposable x86-64 Linux with a kernel configured for protection keys, a C compiler, and permission to run a process that will intentionally receive SIGSEGV. Do not run this on a production host.

cat > mpk-tripwire.c <<'C'
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <unistd.h>

int main(void) {
  setbuf(stdout, NULL);
  size_t page = (size_t)getpagesize();
  int *p = mmap(NULL, page, PROT_READ | PROT_WRITE,
                MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
  if (p == MAP_FAILED) return 1;
  *p = 42;

  int key = pkey_alloc(0, 0);
  if (key == -1) { perror("pkey_alloc"); return 2; }
  if (pkey_mprotect(p, page, PROT_READ | PROT_WRITE, key) == -1) {
    perror("pkey_mprotect"); return 3;
  }
  printf("before: %d; key=%d\\n", *p, key);
  if (pkey_set(key, PKEY_DISABLE_ACCESS) == -1) {
    perror("pkey_set"); return 4;
  }
  puts("next read should fault");
  printf("after: %d\\n", *p);
}
C
cc -O2 -Wall mpk-tripwire.c -o mpk-tripwire
./mpk-tripwire
grep -m1 -oE 'pku|ospke' /proc/cpuinfo | sort -u

Expected result: on a supported machine, the program prints before: 42, then terminates with a segmentation fault at the second read. pku indicates CPU support and ospke indicates kernel support; the actual API call is the authoritative test.

Variation 1: replace PKEY_DISABLE_ACCESS with PKEY_DISABLE_WRITE. The read should succeed, while a later *p = 99 should fault. This shows that MPK further restricts ordinary page permissions; it does not replace PROT_READ/PROT_WRITE.

Variation 2: run perf stat -e cache-misses,branch-misses,context-switches ./mpk-tripwire and repeat after adding unrelated CPU load. The counts are diagnostic context, not proof of Spectre resistance.

Production design rules

• Prefer process or VM isolation when the workload's latency and cost budget allows it; use in-process isolation only with an explicit threat model and continuous adversarial testing.
• Layer defenses: rapid V8 patching, software pointer containment, hardware page tripwires, Spectre-aware scheduling and detection, restricted timers, and a fast quarantine path.
• Measure attacker time-to-exfiltration, not just the presence of a side channel. Amplification and remote observation can invalidate a defense that looked adequate at local timing resolution.
• Make security faults observable as security events: tenant, isolate, CPU class, pkey allocation, process migration, and runtime version—without logging the secret that was at risk.
• Re-test after CPU, kernel, V8, JIT, scheduler, WebSocket, or co-location changes. Microarchitectural defenses are unusually sensitive to implementation drift.

The durable mental model is simple: an isolate controls the program's vocabulary, the V8 Sandbox constrains corrupted pointers, MPK can trap many wrong-page accesses, and process boundaries remove the easiest shared address space. None of those statements makes the cache private. When tenants share a CPU, “not readable by the program” and “not inferable by the program” are different security claims.

Primary sources

• Schwarzl et al., Remote-Timer-as-a-Service: Efficient Microarchitectural Leakage in the Cloud with Remote Timers — August 2026 paper, threat model, remote timers, amplification, measured leakage, and mitigations.
• Cloudflare: A revisit of remote Spectre attacks on Cloudflare Workers — production context, attack primitives, and the statement that the demonstrated attack was mitigated.
• V8: The V8 Sandbox — software containment of corrupted V8 heap memory, pointer compression, offsets, and external-pointer indirection.
• V8 source: external-pointer-table.h — open-source table entry fields, tags, marking, invalidation, and compaction operations.
• Linux man-pages: pkeys(7) — page tagging, per-thread rights, key limits, faults, and MPK caveats.
• Kocher et al., Spectre Attacks: Exploiting Speculative Execution — original transient-execution threat model and cache-observation technique.