---
title: Google Cloud Stateless Inference
framework: Private Cloud Compute Security Guide
role: article
role_heading: Article
path: private-cloud-compute/googlecloudstatelessinference
---

# Google Cloud Stateless Inference

A node serves foundation models on its GPUs while isolating each request, recycling inference processes, and encrypting cached state.

## Overview

Overview The Inference Engine (TIE) handles the application-level logic of executing inference requests on a node, dispatching model execution to the AFM Runtime on the node’s GPUs. TIE manages prompt tokenization, media processing, model metadata, and per-request isolation. Incoming requests enter the node through CloudBoard, which On-Node Request Termination with the user’s device. To decrypt a request, a cb_jobhelper instance passes the client’s key-encapsulation ciphertext to the Protecting request keys through the Appendix: Google Cloud Workload Services Daemon; the Key Protection Service decapsulates the shared secret inside its own confidential VM and returns it sealed to the Workload VM’s binding key, keeping the long-lived private key out of the VM that parses the request. cb_jobhelper then hands the decrypted request over a Unix domain socket to tie-cloud-app, the per-request front end of TIE.

The inference pipeline A node separates the work of serving a model into a per-request application layer, a shared long-lived inference back end, and supporting supervisor daemons: Per-request isolation and statelessness Two mechanisms enforce PCC’s Stateless Computation: per-request process isolation for parsing untrusted input, and periodic recycling of the AFM Runtime. Parsing of the request and rendering of the prompt run in an ephemeral, per-request instance of tie-cloud-app. Within the Megacontainer, systemd launches each request in an Per-request isolation activated over a Unix domain socket from cb_jobhelper. Tearing down this per-request context when the request completes discards intermediate request data, helping ensure that a compromise during input parsing cannot reach another user’s request data. The AFM Runtime is a shared, long-lived process: it maps the model weights in GPU memory and serves multiple requests in succession, rather than being spawned per request. To bound how long private data can persist in memory, tie-controllerd periodically recycles the runtime. Recycling Recycling clears private data that could accumulate in the AFM Runtime, helping ensure that data processed for an earlier request is inaccessible to an attacker who compromises a later instance. The controller daemon (tie-controllerd) periodically terminates and restarts the runtime to bound that accumulation. tie-controllerd triggers a recycle when: A security-driven timeout expires, so private data does not accumulate in the runtime indefinitely. This recycle interval is based on elapsed wall-clock time, and each node draws its own jitter, so a fleet that booted together does not recycle in lockstep. A node enters an error state, triggering an out-of-cycle recycle to return the runtime to health. A model or prompt asset update arrives, or an operator requests maintenance through tie-controllerd’s control interface. tie-controllerd coordinates the recycle so that in-flight requests finish cleanly and no traffic routes to an unready node: tie-controllerd marks the node busy in the routing layer through CloudBoard, preventing new traffic from reaching the node. tie-controllerd calls the Recycle RPC on tie-status-probe, which signals the AFM Runtime to drain in-flight inference requests before the container shuts down. tie-controllerd waits for systemd to start a fresh afm-runtime container, and for the runtime to reattach to the model weights and prefix cache in tie-model-owner and to pass readiness checks. tie-controllerd marks the node available in CloudBoard to resume taking traffic. tie-controllerd queries runtime status (initializing, ready, in error, or recycling) through tie-status-probe over gRPC. It continuously monitors the health of the AFM Runtime, tie-model-owner, and gpuhealthd, reporting a summarized health state to CloudBoard so that node availability reflects the operational state of the inference stack. Recycling tears down the AFM Runtime’s process state and memory. It preserves the model weights and system-prompt prefix cache held by tie-model-owner, allowing the restarted runtime to resume serving without reloading weights or recomputing static caches. Cache privacy The inference stack maintains two caches: a system-prompt prefix cache for shared, static prompt material, and an encrypted per-user key-value (KV) cache that preserves Multi-turn interactions. The confidential VM holds cached state in CPU and GPU memory, with no persistent storage attached. System-prompt prefix cache Recomputing the key-value cache for system prompts on every request is redundant, because the prompt prefix is identical across requests. To avoid recomputation, the AFM Runtime computes the key-value cache for system prompts in advance and stores the resulting blocks in a prefix trie — a form of prefix caching restricted to prompt material supplied by Apple infrastructure, not user input. Because the AFM Runtime uses paged attention, each trie node represents a block of tokens rather than a single token. The prefix cache accelerates both one-shot and multi-turn requests. System prompts are registered offline, and tie-controllerd triggers cache warmup before the node advertises readiness to serve traffic. Because the prefix cache contains only static infrastructure prompts and no user data, it is stored in cleartext. The prefix cache occupies two stores. The long-lived tie-model-owner process owns the primary store in CPU DRAM, and the AFM Runtime holds a read-through copy of hot blocks in GPU high-bandwidth memory (HBM). The CPU store is the source of truth and survives an AFM Runtime recycle; the GPU copy is discarded with the runtime. The prefix cache life cycle has three phases:

Before restoring a cached prefix, the AFM Runtime validates the state reported by tie-model-owner. Restoration proceeds only if every GPU rank reports a ready state and all ranks agree on trie metadata, block count, block size, and physical memory layout. If any rank reports an incomplete or inconsistent state, the runtime discards the cached state across all ranks, resets tie-model-owner, and rebuilds the cache through a fresh warmup sequence. If validation fails after the runtime attaches the persistent pool, the runtime unwinds the memory mappings and halts initialization rather than serving requests from an inconsistent cache. Updates to system prompts arrive through the node’s asset-delivery path; rather than mutating the sealed pool in place, an asset update triggers a recycle that resets the prefix cache and warms it again. Per-user KV cache For multi-turn interactions, TIE reuses the KV cache produced during earlier turns of a conversation, avoiding recomputation of the dialogue history. To preserve statelessness, TIE encrypts cached conversation state with a turn key supplied by the user’s device. At the end of an inference turn, tie-cloud-app encrypts the user’s KV cache and writes it to the node’s cache store, which drops entries when their time-to-live expires and evicts the oldest writes when it runs short of capacity. A turn key from the user’s device makes a cache entry reusable by a later turn. For a request that arrives without one, tie-cloud-app caches the KV cache under a key derived from that request’s own Data Encryption Key, which is reclaimed with the request, so the node can decrypt that entry only while the request is in flight; moving the KV cache between processes covers that derivation. When a subsequent turn of a conversation arrives, tie-cloud-app compares the cached token sequence against the current turn’s prompt, reuses matching cache entries, and instructs the AFM Runtime to evaluate only the divergent remainder. A device-supplied turn key preserves statelessness across turns: once a turn finishes and its ephemeral keys are discarded from memory, any cached state remaining on the node cannot be decrypted by any local process. Continuing the conversation requires the user’s device to supply the key again alongside the subsequent turn’s request. Per-user KV caches persist across AFM Runtime recycles because they reside in the node’s cache store, not inside the runtime process. Nothing written inside the confidential VM outlives it: the VM’s Host and guest OS hardening is encrypted under a key the node generates at boot and never writes to persistent storage. The node stages temporary data, such as downloaded model assets on their way to the GPUs, on in-memory file systems. Cached state resides in volatile CPU and GPU memory and clears when the virtual machine terminates. Moving the KV cache between processes Reusing a conversation’s KV cache across multiple turns requires moving cached state out of GPU memory. GPU memory is reserved for active inference batches and cannot hold idle conversation history. To preserve statelessness, the node exports cached state from the GPU, encrypts it, and places it in the node’s cache store, an in-memory store that retains it until the user’s device initiates the next turn, purging it earlier when a time-to-live expires or when the store comes under capacity pressure. The kvxfer transfer library moves KV cache buffers between the AFM Runtime and tie-cloud-app without copying data. kvxfer allocates anonymous shared memory (memfd) and passes the file descriptor between processes over a Unix domain socket. kvxfer manages buffer headers, block layout, and synchronization, but does not interpret or alter the KV payload. kvxfer synchronizes buffer access: tie-cloud-app reads a block only after the AFM Runtime commits it, committed blocks cannot be overwritten, and an abnormal process exit wakes the waiting consumer with an error rather than leaving it to hang indefinitely. tie-cloud-app embeds EnsembleWarden as a library to encrypt and decrypt the cache in-process: A multi-turn request carries a turn key from the user’s device, and that key encrypts the cache. For a request with no turn key, the node derives an ephemeral key from the request’s Application protocol under a distinct derivation label, so it too exists only for the life of the request. If neither key is available, the node skips caching rather than writing the blocks unencrypted. note: EnsembleWarden