Google Cloud Stateless Inference

A node serves foundation models on its GPUs while isolating each request, recycling inference processes, and encrypting cached state.

Overview

The Inference Engine (TIE) handles the application-level logic of executing inference requests on a node, dispatching model execution to the AFM Runtime on the node’s GPUs. TIE manages prompt tokenization, media processing, model metadata, and per-request isolation.

Incoming requests enter the node through CloudBoard, which On-Node Request Termination with the user’s device. To decrypt a request, a cb_jobhelper instance passes the client’s key-encapsulation ciphertext to the Protecting request keys through the Appendix: Google Cloud Workload Services Daemon; the Key Protection Service decapsulates the shared secret inside its own confidential VM and returns it sealed to the Workload VM’s binding key, keeping the long-lived private key out of the VM that parses the request. cb_jobhelper then hands the decrypted request over a Unix domain socket to tie-cloud-app, the per-request front end of TIE.

[Image]

The inference pipeline

A node separates the work of serving a model into a per-request application layer, a shared long-lived inference back end, and supporting supervisor daemons:

Inference application (tie-cloud-app): The request-handling front end of the pipeline. When cb_jobhelper receives a request, systemd socket-activates a per-request instance of tie-cloud-app in an isolated container. The application tokenizes the prompt and passes any attached image to the AFM Runtime as opaque bytes rather than decoding it. It translates sampling parameters and tool constraints into the grammar used for constrained decoding, dispatches the assembled request to the AFM Runtime over Triton’s gRPC inference API, and detokenizes the tokens streaming back, padding each streamed text delta so that its serialized size falls into one of a fixed set of buckets. Running prompt handling in an ephemeral, per-request context helps ensure that a compromise there cannot reach another user’s request data.
AFM Runtime: The runtime that executes both the language model and the vision encoder. Hosted as a Python back end inside NVIDIA’s Triton Inference Server, it receives inference requests from tie-cloud-app over Triton’s gRPC inference API and streams generated tokens back. The vision encoder decodes an attached image before encoding the pixels. The decoder is written in Rust with no C codec beneath it, and fails any request carrying an image in a format it doesn’t decode, including HEIC and AVIF. Still images are the only compressed media a node decodes. Decoding runs in the shared runtime instead of a per-request process, so periodic recycling bounds how long a compromise of the decoder can persist. The runtime manages continuous batching of incoming requests, coordinates the paged KV cache, enforces grammars during token sampling, and evaluates stop conditions. It divides the language model across the node’s GPUs using tensor and pipeline parallelism.
Bones: The library that executes the language model for the AFM Runtime: the transformer layers and the CUDA kernels beneath them. Bones is written in Rust for NVIDIA Blackwell GPUs; the AFM Runtime embeds it through Python bindings and drives one forward pass per scheduled batch.
Controller (tie-controllerd): The long-lived daemon that supervises the inference stack. It monitors the health of the AFM Runtime and the GPUs, coordinates periodic recycling, and registers the node’s availability and active workload with CloudBoard, which publishes node readiness to the routing layer.
Status probe (tie-status-probe): The AFM Runtime’s health and recycle interface. It runs inside the afm-runtime container as a critical process supervised by supervisord, and exposes a gRPC interface to tie-controllerd for health checks and recycle requests. tie-status-probe verifies readiness by executing a canary streaming inference of a single token with an empty grammar, exercising and initializing the streaming and constrained-decoding pipelines before the node reports itself ready. When tie-controllerd requests a recycle, tie-status-probe initiates a graceful shutdown. The probe is a critical process, so a wrapper script shuts supervisord down as soon as the probe exits, bringing the entire afm-runtime container down so that systemd can create a fresh one.
GPU health daemon (gpuhealthd): The controller’s source of truth for the GPUs. It polls each GPU through the NVIDIA Management Library (NVML) for telemetry and fault conditions, classifies each GPU as healthy, degraded, or unavailable, and serves that verdict to tie-controllerd over a local Unix domain socket. tie-controllerd requires a healthy verdict from gpuhealthd before advertising the node as ready to take traffic, and treats a subsequent failure as an error state.
Model owner (tie-model-owner): A long-lived daemon that holds the model weights and the system-prompt prefix cache in its own container. Because tie-model-owner remains running across an AFM Runtime recycle, the node does not need to reload weights from disk or recompute prefix caches when the runtime restarts. During initial startup, tie-model-owner reads the model weights into GPU memory and shares them with the AFM Runtime using CUDA IPC memory handles; it allocates the prefix cache in host CPU memory and shares it over a Unix domain socket as a file descriptor. When a recycled AFM Runtime starts, it maps these existing memory references directly.

Per-request isolation and statelessness

Two mechanisms enforce PCC’s Stateless Computation: per-request process isolation for parsing untrusted input, and periodic recycling of the AFM Runtime.

Parsing of the request and rendering of the prompt run in an ephemeral, per-request instance of tie-cloud-app. Within the Megacontainer, systemd launches each request in an Per-request isolation activated over a Unix domain socket from cb_jobhelper. Tearing down this per-request context when the request completes discards intermediate request data, helping ensure that a compromise during input parsing cannot reach another user’s request data.

The AFM Runtime is a shared, long-lived process: it maps the model weights in GPU memory and serves multiple requests in succession, rather than being spawned per request. To bound how long private data can persist in memory, tie-controllerd periodically recycles the runtime.

Recycling

Recycling clears private data that could accumulate in the AFM Runtime, helping ensure that data processed for an earlier request is inaccessible to an attacker who compromises a later instance. The controller daemon (tie-controllerd) periodically terminates and restarts the runtime to bound that accumulation.

tie-controllerd triggers a recycle when:

A security-driven timeout expires, so private data does not accumulate in the runtime indefinitely. This recycle interval is based on elapsed wall-clock time, and each node draws its own jitter, so a fleet that booted together does not recycle in lockstep.
A node enters an error state, triggering an out-of-cycle recycle to return the runtime to health.
A model or prompt asset update arrives, or an operator requests maintenance through tie-controllerd’s control interface.

tie-controllerd coordinates the recycle so that in-flight requests finish cleanly and no traffic routes to an unready node:

tie-controllerd marks the node busy in the routing layer through CloudBoard, preventing new traffic from reaching the node.
tie-controllerd calls the Recycle RPC on tie-status-probe, which signals the AFM Runtime to drain in-flight inference requests before the container shuts down.
tie-controllerd waits for systemd to start a fresh afm-runtime container, and for the runtime to reattach to the model weights and prefix cache in tie-model-owner and to pass readiness checks.
tie-controllerd marks the node available in CloudBoard to resume taking traffic.

tie-controllerd queries runtime status (initializing, ready, in error, or recycling) through tie-status-probe over gRPC. It continuously monitors the health of the AFM Runtime, tie-model-owner, and gpuhealthd, reporting a summarized health state to CloudBoard so that node availability reflects the operational state of the inference stack.

Recycling tears down the AFM Runtime’s process state and memory. It preserves the model weights and system-prompt prefix cache held by tie-model-owner, allowing the restarted runtime to resume serving without reloading weights or recomputing static caches.

Cache privacy

The inference stack maintains two caches: a system-prompt prefix cache for shared, static prompt material, and an encrypted per-user key-value (KV) cache that preserves Multi-turn interactions. The confidential VM holds cached state in CPU and GPU memory, with no persistent storage attached.

System-prompt prefix cache

Recomputing the key-value cache for system prompts on every request is redundant, because the prompt prefix is identical across requests. To avoid recomputation, the AFM Runtime computes the key-value cache for system prompts in advance and stores the resulting blocks in a prefix trie — a form of prefix caching restricted to prompt material supplied by Apple infrastructure, not user input. Because the AFM Runtime uses paged attention, each trie node represents a block of tokens rather than a single token. The prefix cache accelerates both one-shot and multi-turn requests. System prompts are registered offline, and tie-controllerd triggers cache warmup before the node advertises readiness to serve traffic. Because the prefix cache contains only static infrastructure prompts and no user data, it is stored in cleartext.

The prefix cache occupies two stores. The long-lived tie-model-owner process owns the primary store in CPU DRAM, and the AFM Runtime holds a read-through copy of hot blocks in GPU high-bandwidth memory (HBM). The CPU store is the source of truth and survives an AFM Runtime recycle; the GPU copy is discarded with the runtime.

The prefix cache life cycle has three phases:

Warmup: tie-controllerd sends the registered system-prompt requests. The AFM Runtime computes their KV blocks on the GPUs, constructs the prefix trie, and transfers the block data to the CPU memory pool in tie-model-owner. Once warmup completes, the runtime serializes the trie metadata to tie-model-owner and seals the pool. The Linux kernel enforces the sealing with file seals (F_SEAL_FUTURE_WRITE, F_SEAL_SHRINK, F_SEAL_GROW, and F_SEAL_SEAL) on the underlying anonymous shared memory descriptor (memfd), preventing subsequent writable mappings and fixing the pool’s size. An AFM Runtime that restores the pool maps it read-only, and the seal stops it from taking a writable mapping. The instance that warmed the pool keeps the writable mapping it allocated, and stops writing to the pool after sealing it.
Serving: Incoming requests match against the prefix trie. The AFM Runtime serves matched blocks from GPU HBM or loads them from the CPU memory pool in tie-model-owner on an HBM miss. The model’s forward pass then evaluates only the user tokens that follow the matched prefix. The sealed CPU memory pool is read-only during serving.
Recycle: When a fresh AFM Runtime starts after a recycle, it retrieves the trie metadata and memory pool descriptor from tie-model-owner, rebuilds the trie structure in memory without evaluating prompts through the GPUs, and allocates a fresh GPU HBM pool. Initial requests reload blocks from the CPU memory pool on demand, avoiding startup warmup delay.

[Image]

Before restoring a cached prefix, the AFM Runtime validates the state reported by tie-model-owner. Restoration proceeds only if every GPU rank reports a ready state and all ranks agree on trie metadata, block count, block size, and physical memory layout. If any rank reports an incomplete or inconsistent state, the runtime discards the cached state across all ranks, resets tie-model-owner, and rebuilds the cache through a fresh warmup sequence. If validation fails after the runtime attaches the persistent pool, the runtime unwinds the memory mappings and halts initialization rather than serving requests from an inconsistent cache. Updates to system prompts arrive through the node’s asset-delivery path; rather than mutating the sealed pool in place, an asset update triggers a recycle that resets the prefix cache and warms it again.

Per-user KV cache

For multi-turn interactions, TIE reuses the KV cache produced during earlier turns of a conversation, avoiding recomputation of the dialogue history. To preserve statelessness, TIE encrypts cached conversation state with a turn key supplied by the user’s device. At the end of an inference turn, tie-cloud-app encrypts the user’s KV cache and writes it to the node’s cache store, which drops entries when their time-to-live expires and evicts the oldest writes when it runs short of capacity. A turn key from the user’s device makes a cache entry reusable by a later turn. For a request that arrives without one, tie-cloud-app caches the KV cache under a key derived from that request’s own Data Encryption Key, which is reclaimed with the request, so the node can decrypt that entry only while the request is in flight; moving the KV cache between processes covers that derivation. When a subsequent turn of a conversation arrives, tie-cloud-app compares the cached token sequence against the current turn’s prompt, reuses matching cache entries, and instructs the AFM Runtime to evaluate only the divergent remainder.

A device-supplied turn key preserves statelessness across turns: once a turn finishes and its ephemeral keys are discarded from memory, any cached state remaining on the node cannot be decrypted by any local process. Continuing the conversation requires the user’s device to supply the key again alongside the subsequent turn’s request. Per-user KV caches persist across AFM Runtime recycles because they reside in the node’s cache store, not inside the runtime process.

Nothing written inside the confidential VM outlives it: the VM’s Host and guest OS hardening is encrypted under a key the node generates at boot and never writes to persistent storage. The node stages temporary data, such as downloaded model assets on their way to the GPUs, on in-memory file systems. Cached state resides in volatile CPU and GPU memory and clears when the virtual machine terminates.

Moving the KV cache between processes

Reusing a conversation’s KV cache across multiple turns requires moving cached state out of GPU memory. GPU memory is reserved for active inference batches and cannot hold idle conversation history. To preserve statelessness, the node exports cached state from the GPU, encrypts it, and places it in the node’s cache store, an in-memory store that retains it until the user’s device initiates the next turn, purging it earlier when a time-to-live expires or when the store comes under capacity pressure.

The kvxfer transfer library moves KV cache buffers between the AFM Runtime and tie-cloud-app without copying data. kvxfer allocates anonymous shared memory (memfd) and passes the file descriptor between processes over a Unix domain socket. kvxfer manages buffer headers, block layout, and synchronization, but does not interpret or alter the KV payload. kvxfer synchronizes buffer access: tie-cloud-app reads a block only after the AFM Runtime commits it, committed blocks cannot be overwritten, and an abnormal process exit wakes the waiting consumer with an error rather than leaving it to hang indefinitely.

tie-cloud-app embeds EnsembleWarden as a library to encrypt and decrypt the cache in-process:

Exporting cache: At the end of an inference turn, the AFM Runtime copies KV cache blocks from GPU memory into a shared kvxfer buffer. tie-cloud-app reads the buffer, encrypts the cache blocks with EnsembleWarden using the user’s turn key, and writes the result to the node’s cache store for subsequent turns.
Importing cache: At the start of a subsequent turn, tie-cloud-app reads the encrypted cache from the node’s cache store and decrypts it with EnsembleWarden into a shared kvxfer buffer. The AFM Runtime validates the buffer’s metadata (model identifier, release version, block size, and token count) against its active configuration before scattering the decrypted cache blocks into GPU memory. If validation fails or decryption fails, the runtime aborts the restore and evaluates the prompt without the cached history.

A multi-turn request carries a turn key from the user’s device, and that key encrypts the cache. For a request with no turn key, the node derives an ephemeral key from the request’s Application protocol under a distinct derivation label, so it too exists only for the life of the request. If neither key is available, the node skips caching rather than writing the blocks unencrypted.