--- title: On-Node Request Termination framework: Private Cloud Compute Security Guide role: article role_heading: Article path: private-cloud-compute/requesttermination --- # On-Node Request Termination A PCC node terminates and decrypts each request with attested keys in a per-request context that the node then tears down when the request completes. ## Overview Overview A request arrives at a PCC node encrypted to the node’s attested public key. The sending device has Node validation and wrapped the request’s keys to that node, so the node that receives it is one the device chose to trust. On the node, decrypted user data and attacker-controlled input exist in the same place at the same time: the node decrypts the request, and the code that parses its contents is the first PCC code to operate on input from outside the High-security inference. Two goals shape the node’s design: Bound what a compromise at that moment reaches: subverting the code that handles one request should not give an attacker access to another user’s data. Enforce Stateless computation on personal user data: when a request completes, neither its data nor the ability to decrypt it survives on the node. CloudBoard Every PCC node runs CloudBoard, a collection of daemons that manage the node’s request life cycle and stand between the network and the PCC cloud applications that services the request. CloudBoard owns the two ends of that life cycle: it publishes the node’s availability so that routing can reach it, and it terminates each incoming request, decrypting the request stream before any application code sees it and encrypting the responses that go back. Before CloudBoard advertises a node as available, the workload the node serves must be healthy and ready to serve. The workload’s controller reports its health to CloudBoard, and CloudBoard in turn registers the node with the fleet’s routing and service-discovery layer, along with information about the workload it can serve, such as the model an inference node loaded and the metadata that routing uses to match requests. Routing can direct a request to the node only after this registration. Because CloudBoard accepts incoming network connections, it sits on an attack surface that faces outside the PCC trust boundary. cloudboardd is the network-facing daemon, and the central coordinator that manages all interaction with the routing layer: it exposes the node’s status and health, publishes the attestations it receives from cb_attestationd, and accepts incoming requests. It holds no keys that can decrypt request data, so terminating a connection and decrypting what arrives on it happen in separate processes. Each time cloudboardd accepts a request, it assigns a dedicated cb_jobhelper instance to manage that request, proxying the request’s network stream between the routing layer and that helper. The handshake with the client and the decryption of the request stream happen there, in a context created for that one request. note: CloudBoard: the Apple silicon implementation CloudBoard-GoogleCloud: the Google Cloud implementation Per-request isolation CloudBoard binds each request to a fresh execution context that exists only for the lifetime of that one request: the cb_jobhelper instance that terminates the request, paired with a per-request instance of the cloud application that services it. A node can handle many requests concurrently, each in its own context. The initial parsing of the request’s contents runs inside this context. When the request completes, CloudBoard tears the context down, terminating cb_jobhelper and the instance of the cloud application tied to that request. What isolates one context from the next differs by platform: Because the operating system reclaims the context rather than the application code discarding its contents, intermediate request data, per-request state, and request-decryption keys all go with it. This design helps ensure that a compromise of the code that terminates and parses one request doesn’t expose another user’s data. Decrypting the request The device encrypted the request to the key presented by the node’s attestation. PCC is designed to ensure that the private half of that key is not exportable: cb_attestationd creates the Application protocol (REK) inside a boundary that performs operations with it but does not release it, so the code that handles a request never holds the key itself, only a reference to it and the secret negotiated for the request it is serving. This arrangement provides two properties: An attacker who compromises the code that terminates and parses requests can read the requests that the node is handling for as long as the compromise lasts, but cannot extract the node’s key to decrypt traffic collected elsewhere, and the exposure ends when the key expires. The node doesn’t retain anything it can decrypt once the request completes, because the request’s secret lives only in the per-request context. The platforms differ in what provides that boundary and in how the key stays tied to the state the device validated. The cloud app parses untrusted contents, so it is the likeliest place for an attacker to gain execution on a node. Per-request isolation contains such a compromise to the data that one process holds, and PCC is designed to keep the request-decryption key out of that process. When a Multi-turn interactions reuses cached KV state across turns, the device supplies encryption keys to protect that cached state while it is stored. Apple silicon cb_attestationd creates the REK inside the Apple Silicon Root of Trust, which generates the private half and never releases it. The Secure Enclave binds the key to the OS instance and to the node’s Software Sealed Registers, the ratcheted measurements of the cryptexes and configuration the node loaded after boot. Because the Secure Enclave entangles that register state with the key, the key becomes inaccessible if the measurements change: a change to the node’s software invalidates the key rather than carrying it into a state the device never validated. cb_attestationd hands cloudboardd the The attestation bundle covering the key, together with the key’s identifier and expiry, and cloudboardd publishes those to the routing layer for devices to validate; nothing in that handoff can use the key. A cb_jobhelper instance instead receives a reference to the key, which it resolves to perform the Hybrid Public Key Encryption handshake with the client while the private-key operation executes inside the Secure Enclave. What comes back is the client’s Application protocol (DEK). cb_jobhelper uses the DEK to decrypt the streamed request, and the DEK is reclaimed with the per-request context. cb_jobhelper can start one or more support apps alongside the cloud app, tied to the same request lifetime. A support app is not on the data path for application-layer protocol messages, which pass only between cb_jobhelper and the cloud app; it carries narrow responsibilities that isolate security-critical cryptographic functionality from the application logic, such as managing application-specific keys. The protocol between the client and cb_jobhelper can therefore carry crypto material alongside the application-layer messages: the messages are forwarded to the cloud app, while any accompanying crypto material is released only to a support app that holds the dedicated com.apple.private.cloudos.cb_jobhelper.crypto_material.access entitlement. The The Inference Engine configures EnsemblePorter as its support app, so ensembleporterd holds the device’s KV cache reuse. Google Cloud On a Google Cloud node, the boundary that holds request keys is a Confidential computing (CVM), a virtual machine whose memory the CPU’s confidential-computing hardware isolates from the host. The node runs the Key Protection Service in Protecting request keys, separate from the Workload VM that parses each request’s untrusted input, so the component exposed to a request’s contents never holds the key that decrypts it. cb_attestationd asks the Key Protection Service to generate the key, reaching it through the Appendix: Google Cloud Workload Services Daemon; the private half stays in that CVM, and cb_attestationd receives only a handle to the key and the attestation bundle covering it. When a request arrives, the node doesn’t unseal a key locally. The cb_jobhelper instance passes the client’s key-encapsulation ciphertext, together with the key’s handle, to the Key Protection Service through the Workload Services Daemon. The Key Protection Service decapsulates the shared secret inside its own boundary and returns it sealed to a binding key that only the Workload VM can open, so the secret never passes between the two CVMs in the clear and the long-lived private key never leaves the Key Protection Service. cb_jobhelper then decrypts the request with the released secret. The key is tied to the node’s attested state at the point it is issued: a verifier checks the Attested key release of the Key Protection Service and the Workload VM before any device encrypts to the key, and the key then carries a bounded lifetime, after which the Key Protection Service deletes it. On a Google Cloud node, cb_jobhelper forwards the per-turn cache keys alongside the request directly into the cloud application, which encrypts the cache itself rather than delegating the encryption to a support app. The request-decryption key itself remains isolated in cb_jobhelper. Rejecting replayed requests An attacker who captures a request in flight and submits it again accumulates observations of the same computation; each additional observation extends what a Side-channel extraction can recover. To mitigate this, cloudboardd rejects a request it has already seen. cloudboardd derives the anti-replay identifier from the client’s key exchange: the encapsulated key material a client sends is unique to that request, so the daemon keeps a set of fixed-size identifiers, one for each piece of key material it has accepted under each of the node’s current request-encryption keys, and refuses a request whose identifier is already in that set. The check runs on the message that contains the key material, which arrives before any request body, so a rejected replay never reaches a cloud app. The client receives a permission error that identifies the request as a suspected replay. Two properties limit the set of identifiers a node must check: A request’s key identifier is specific to the node that published it, so a request replayed to a different node fails decapsulation there regardless of what that node has seen. The set has to cover only a request-encryption key’s lifetime: once a key expires the node deletes its private half, so nothing can decrypt a request encrypted to a retired key. Each platform has its own approach to managing this set: Handoff to the application Once the node decrypts the request, the per-request context passes the decrypted request to the application, which deserializes and validates the request’s contents. cb_jobhelper encrypts the response messages the application returns and forwards them to cloudboardd, which proxies the stream back toward the client over the end-to-end encrypted channel established during routing. A node serving foundation-model inference passes the validated request on to its inference stack, where the model runs on the node’s accelerators. A foundation model is far too expensive to load per request, so the software that serves it is shared and long-lived rather than torn down with the request; it is instead periodically recycled to clear private data left behind by the requests it has served, on The Inference Engine as on Recycling. A request that a single node cannot service on its own is distributed to further nodes: a proxy node validates each downstream node’s attestation on the device’s behalf before forwarding the request to that node, extending the device’s single trust decision across the set of nodes that cooperate on one request through Transitive Trust. Outbound requests to external services While handling a user’s request, some applications make an outbound request to a service outside the PCC node, such as a Private Information Retrieval query or an External tool calls. PCC’s approach to supporting these calls addresses two constraints: Processes that hold decrypted user data must stay isolated from the external network, so a compromise of the code that parses a request cannot reach out from the node directly. Data that crosses the PCC trust boundary must be privacy-preserving in its own right: private information retrieval, for example, uses homomorphic encryption so the external server computes on the query without being able to decrypt it. To satisfy both, a dedicated daemon, cb_bridged, handles all outbound requests that carry request data. The cloud application or cb_jobhelper sends request data to cb_bridged over XPC; cb_bridged performs the network request and returns the response. This approach keeps the node’s outbound path for user data in one narrow component that never holds a key to a user’s request.