---
title: Apple Silicon Stateless Inference
framework: Private Cloud Compute Security Guide
role: article
role_heading: Article
path: private-cloud-compute/statelessinference
---

# Apple Silicon Stateless Inference

A node’s inference engine enables high performance while keeping user data private.

## Overview

Overview To power LLM functionality in PCC, we designed a custom inference server — The Inference Engine (TIE) — and inference framework — MetalLM. MetalLM uses Metal-based shaders and compute kernels to perform inference computation. TIE and MetalLM also support distributing inference across multiple SoCs to process requests with reduced latency and higher throughput. PCC supports distributed inference across a grouping of up to 32 nodes, together called an ensemble, with a leader node that’s responsible for accepting incoming requests and coordinating the ensemble’s activities. Language model inference We designed foundation language models for Apple Intelligence to perform a wide range of tasks efficiently, accurately, and responsibly. This includes Apple Foundation Model server (AFM-server), the model used by PCC. The Apple Intelligence Foundation Language Models paper and our blog post Introducing Apple’s On-Device and Server Foundation Models describe the architecture and training of AFM-server. Notable aspects of the inference process include: The model can be specialized for specific tasks using runtime-swappable adapters, which are small neural network modules that plug into the base model. The user’s device submits inference requests by providing values which TIE composes with a prompt template to produce the model’s prompt. These values consist of text strings and optionally an image. The model uses a tokenizer based on SentencePiece. TIE is responsible for tokenizing the string prompt and converting output tokens back into strings. The model uses an image tokenizer trained for generating image embeddings if there is an image in the prompt. TIE might leverage speculative decoding using a small draft model to speed up inference. Some use cases may prescribe a grammar for the output, in which case a framework for constrained decoding helps ensure that sampled tokens meet the prescribed grammar. Model output is streamed to the user’s device as tokens are produced. Padding is applied to prevent token-length side-channel attacks. The Inference Engine TIE is a PCC PCC cloud applications that handles the application-level logic of executing the inference requests that users’ devices submit to PCC. Requests are initially received on the PCC node by CloudBoard, which is responsible for implementing the cryptographic protocol with a user’s device, and are then handed off to TIE for processing. Both TIE and CloudBoard use a family of daemons to ensure robustness and provide isolation in the event of an exploit.

When the system boots, the tie-model-owner process uses the ModelCatalogSE framework to load the model weights, adapters, and associated parameters from disk into memory. ModelCatalogSE provides an index of model data and metadata — for example, tokenizers, stop tokens, and prompt templates — derived from Foundation model and adapter cryptexes on the PCC node. The long-lived tie-model-owner process creates and vends persistent, read-only memory references for the model weights to avoid reloading these large assets from disk. The inference components are orchestrated by a single tie-controllerd daemon, which executes as part of node initialization and coordinates the remaining processes. The tie-controllerd daemon acts as the main control system of TIE and handles interfacing with CloudBoard to provide real-time updates on the health of the system. tie-controllerd also coordinates the recycling of the inference process in addition to initiating periodic key rotations of the CIO mesh, which we discuss below. Lastly, tie-controllerd interfaces with CloudBoard’s CloudConfigurationPreferences framework to fetch prompt deny lists, which are lists of blocked inputs that might compromise system stability. When a request arrives at a PCC node, TIE decodes and pre-processes it in an ephemeral, per-request process: an instance of tie-cloud-app. The system uses process pooling to avoid the runtime cost of process spawning. Process pools consist of pre-instantiated cb_jobhelper and tie-cloud-app pairs that stand ready to serve a request. Once CloudBoard selects a tie-cloud-app instance to handle a request, tie-cloud-app: Deserializes the incoming Protobuf-encoded request Validates that the request parameters are safe Tokenizes the string-based prompt component using the tokenizer specified by the model cryptex Decodes the image in the prompt component using the memory-safe, Swift-based JPEG image parser TIE performs tokenization in a per-request process to isolate parsing of untrusted text and image input. Using a per-request process instance helps to ensure that if a process compromise were to occur, it would not provide access to the request data of other users. TIE discards intermediate request data when the request completes and the process terminates. Upon initial processing of the request, the tie-cloud-app sends the request to a shared tie-inference process that performs the inference using MetalLM. The request data includes the tokenized input, the decoded image if an image is part of the prompt, the identifiers of the model and adapter to use for inference, sampling parameters, and, if applicable, the constrained decoding grammar. To minimize the risk that the inference host process unexpectedly retains user data beyond the lifetime of a single request, the tie-controllerd periodically terminates the inference process and starts a new instance. This approach helps ensure that data processed within a previous instance of the inference host is inaccessible to an attack that might compromise a future instance. Some Apple Intelligence features expect both text and image inputs to generate a response. On the device side, image data is sent to the TrustedMLClient in raw RGB format, which is then encoded to JPEG format with width and height as the only metadata and sent to the server. Upon receiving image data on the server side, tie-cloud-app parses the image using an internally developed memory-safe, Swift-based JPEG decoder. The tokenized text prompt and parsed image data are then interleaved within tie-cloud-app, following what is specified in the prompt template, before it is sent to tie-inference for processing. At tie-inference, the image input is first tokenized into embeddings using the image tokenizer, then fused with text embeddings to generate the final response tokens from the LLM model. Some Apple Intelligence features expect the LLM model to produce output in a specific format. To enable the model to match the expected output format, TIE supports using a constrained decoding process provided by the TokenGenerationSE framework. This imposes constraints on the sequence of tokens that TIE can generate, specified using a BNF-based grammar or JSON Schema that is contained in the model cryptex. For use cases that require dynamic constraints, the client can provide dynamic fragments of grammar in the request body that are joined with the static portion in the model cryptex. The initial processing of this client-provided grammar takes place in tie-cloud-app before the parsed grammar is passed to tie-inference. Before the MetalLM framework generates each token, TokenGenerationSE generates a mask of valid potential tokens. Tokens that fall outside of this mask are rejected by TIE, ensuring that MetalLM generates valid tokens and text. Static prefix cache Many inference requests share common prefix tokens in the form of system prompts and tool definitions that appear unchanged at the start of every request for a given use case. Recomputing the KV cache for these tokens on every request is redundant work. To avoid it, model and adapter cryptexes may declare prompt token sequences to pre-cache, a form of prefix caching restricted to prompt material supplied by Apple infrastructure rather than to user input. At model load, tie-inference: Tokenizes these entries. Uses MetalLM to evaluate them through the model. Constructs a prefix trie of the resulting KV cache data in shared memory owned by tie-model-owner. MetalLM looks up entries from the trie when servicing each request, so the matching prefix isn’t recomputed. The data fed into the trie comes exclusively from prompt material supplied by Apple infrastructure; user input does not influence which token sequences are pre-cached or what is stored in the trie. For larger adapter sets, the full prefix trie may exceed its allocated memory budget. When this happens, tie-inference splits the trie: a static portion remains resident in shared memory and the overflow is written to a file on the per-boot data volume. Distributed inference Distributing requests across multiple nodes presents some unique challenges: we must not weaken PCC’s Verifiable Transparency promises or extend our trust boundary outside the scope of the inference nodes. The nodes in an ensemble are connected in a mesh, so they can exchange data over a high-performance interconnect using USB4 and, in some cases, Ethernet. Ensembles succeed or fail together, and the loss of any ensemble member requires all nodes to reboot. Ensembles have a leader that coordinates the distribution of work. In turn, TIE and MetalLM divide the LLM computation among nodes in the ensemble. To do this, the AppleCIOMesh framework provides a number of data-distribution primitives (similar to those offered by Message Passing Interface) such as peer-to-peer messages, gather-to-all messages, and broadcast messages. TIE (through the MetalLM framework) sends data that’s fully encrypted by AppleCIOMesh to the appropriate set of nodes within an ensemble to parallelize the LLM computation. When operating in distributed inference mode, TIE splits the input prompt tokens among all nodes in the ensemble to produce the first output token. TIE uses the common technique of a key-value cache (KV cache) to avoid recomputing values for tokens that it has already processed. As computation progresses layer by layer in the model, each node initiates a broadcast-and-gather operation to send its KV cache and in turn receive the KV caches of the remaining nodes in the ensemble. To produce additional tokens, nodes exchange tensors per layer at the start and end of each layer, using broadcast-and-gather operations. The leader node of the ensemble samples tokens, checks for stop sequences, and formats token response messages to the client. The tie-controllerd on the leader node coordinates the follower nodes over a TLS-protected gRPC connection keyed by a pre-shared key. tie-controllerd on the leader node initiates control messages to inform the nodes to start a given inference, check health status, and perform periodic inference-process exits. During each inference-process exit event, tie-controllerd informs ensembled to rotate the underlying encryption key of the CIO mesh. AppleCIOMesh is a new framework and kernel extension that provides a secure and efficient low-latency transport among the PCC nodes in an ensemble. AppleComputeEnsembler supplies the primary seed and configures AppleCIOMesh. AppleCIOMesh secures all data on both USB4 and TCP over Ethernet transports with AES-GCM following best practices from NIST. Data on the transport is encrypted by AppleCIOMesh with AES-128-GCM using a unique symmetric key for each node. The GCM anti-replay value is 96 bits: 64 bits of random data and a 32-bit counter. The 64-bit random data portion of the anti-replay value is unique per node and derived from the primary seed using HKDF. Each invocation of GCM increases the counter, and if the invocation amount reaches 2^32, the system triggers a catastrophic failure. The counter is implicit and not sent over the wire. Any data that fails to decrypt is a catastrophic failure, and the ensemble is taken offline. In addition, the system treats link loss as a catastrophic failure. These failures require a reboot and reconfiguration of the entire ensemble. When used on USB4, each link is point-to-point with a fixed link topology. When used with Ethernet, nodes connect using the hostnames configured in the ensemble config and a fixed port number. They identify themselves with a handshake message that includes the sender ID and a message encrypted using that sender’s encryption key, thereby relying on the authentication properties of this encryption scheme to ensure mutual authentication. note: AppleCIOMesh Ensemble attestation The leader of an ensemble is the only PCC node that is exposed to users’ devices and capable of receiving incoming inference requests. Users’ devices verify only the attestation state of the leader to avoid needing to verify the attestation of every node. Because the device relies on the leader node to transitively attest to the state of the ensemble, the leader should maintain consistency across the ensemble. To do so, the leader node confirms that: Each follower node has a valid attestation, signed by a Cryptographic identity certified by the Apple Silicon Hardware Integrity. The software measurements of all follower nodes are identical to the leader’s software measurements. The chip ID, board ID, SoC fusing bits (production status, security mode, and security domain) of all follower nodes are identical to those of the leader. The states of Restricted Execution Mode, Ephemeral Data Mode, and Developer Mode of all follower nodes are identical to those of the leader. The config-security-policy key within darwin-init for each follower is set to the same value as the leader. A darwin-init entry on the leader defines the ensemble configuration and includes the UDIDs and hostnames of all follower nodes. A daemon called ensembled consumes this configuration and uses CloudAttestation to perform a mutually attested key exchange between the leader and each follower. The DCIK certificate fingerprints for the ensemble are also included in the secure-config section of the attestation bundle, and CloudAttestation accepts a node’s attestation only if the node’s secure-config contains them. Users’ devices can also observe this paper trail of certificate fingerprints when validating the attestation bundles of leader nodes. Because darwin-init applies its configuration once per boot and ensembled reads the ensemble topology once when it starts, the registered ensemble topology remains immutable for the entirety of the boot session. AppleCIOMesh also locks the mesh once the mesh is active. Therefore, new ensemble formations can complete the pairing process only if the nodes are rebooted with the correct darwin-init that contains the new topology. Ensemble nodes can securely exchange messages with each other using Attested TLS, which uses Apple Silicon Node Attestation to construct mutual TLS 1.3 connections backed by SEP-attested identity keys. To establish their X.509 certificate-based identities for mutual TLS connections, each node: Generates a SEP-managed NIST P-384 Elliptic Curve Digital Signature Algorithm (ECDSA) key and accompanying attestation bundle. Uses these components to construct a self-signed X.509 certificate, with the attestation bundle serialized as an X.509v3 extension. Validates its peer’s identity during the CertificateVerify step of the TLS handshake by verifying both the X.509 certificate and the corresponding attestation bundle embedded within it. This provides a trusted DCIK certificate, UDID, and public key of the peer node. The attestation validation process in the ensemble mesh scenario matches how users’ devices validate leader nodes, except that the software measurements are not verified against the transparency log. Instead, all nodes in an ensemble verify that their peers’ measurements match the same PCC release. Because a user’s device validates the leader’s release against the log, this check ties every follower to a logged release as well. To begin the ensemble-pairing process, the leader securely generates a random 16-byte primary seed using CryptoKit. The leader must then securely distribute this seed to each follower. Each node uses this seed as input to HKDF-SHA384 to generate a unique-per-sender AES-128-GCM table of keys to secure communication. Once every follower node has checked in and the leader’s connections to them are established, the leader node distributes the shared secret to each follower node over an Attested TLS connection, so a shared secret seed exists between all members of the ensemble. AppleCIOMesh generates a table of unique-per-sender AES-128-GCM keys, which it computes with HKDF-SHA384 using the shared secret seed as the input key material and the nodes’ position in the ensemble topology as the input salt. note: AppleComputeEnsembler CloudAttestation Ensemble Support CloudAttestation Attested TLS Prefill-decode disaggregation In addition to distributing the work required to produce each output token within an ensemble’s nodes, TIE may split the phases of a request across separate ensembles so that each phase can be handled most efficiently. To do this, the ensembles hand off their in-memory state in the form of the KV cache, in addition to the request.

As discussed in distributed inference, a KV cache is used to accelerate LLM inference by storing intermediate results during token generation. Transferring this cache to a new ensemble allows that ensemble to continue processing the request without recalculating these values. LLM output generation occurs in two distinct phases: Prefill phase: This phase handles the initial processing of input data (such as prompt strings and images) and benefits most from distributed inference. Decode phase: After the prefill phase is complete, output tokens are generated one at a time sequentially. Some PCC code, log entries, and tools (including the VRE inference client) refer to the prefill phase as the first token (FT) phase and to the decode phase as the extend token (ET) phase. To maximize resource efficiency, TIE may split a request across two different ensembles: The prefill phase is executed on a larger ensemble optimized for building and scaling the KV cache efficiently. The cached state is then transferred to a smaller ensemble optimized for sequential output generation during the decode phase. The KV cache data is encrypted prior to transport to help ensure the confidentiality of the user’s request. The cb_jobhelper daemon derives a KV-Cache Data Encryption Key (KVDEK) for the prefill-decode handoff from the Application protocol (DEK) provided by the user’s device, using HKDF-SHA256 with a fixed label that identifies the handoff purpose. Because the KDF output is a deterministic function of the DEK, and the label is fixed, every ensemble that processes the request derives the same KVDEK from its copy of the DEK without exchanging key material, helping to ensure that only ensembles with access to the original DEK are able to decrypt this cached state. The KV cache reuse mechanism for Multi-Turn Agent, described in KV cache reuse below, uses a separate set of per-turn keys generated on the user’s device rather than this DEK-derived KVDEK. EnsemblePorter manages the distribution and life cycle of the KVDEK across all nodes in an ensemble. On the leader node, ensembleporterd receives the KVDEK from cb_jobhelper and stores it for the duration of the request. On each follower node, a corresponding ensemblefollowerporterd process receives and manages the keys. Both processes are per-request and are terminated when the request completes, helping to ensure that key material is automatically discarded from memory. The distribution of keys from the leader to follower nodes uses end-to-end encryption to prevent the long-running ensembled daemon from having access to unencrypted key material. When a new request arrives, ensembleporterd and each ensemblefollowerporterd perform an ephemeral Curve25519 key agreement to establish a per-instance shared secret. The leader then uses this shared secret to encrypt the KVDEK with AES-GCM before sending it through ensembled. Because ensembled only ever sees the encrypted key payload, it cannot access the unencrypted KVDEK — even though it serves as the transport layer between ensemble nodes. Access to the KVDEK is restricted through entitlement-based access control. TIE processes lack entitlements to access the keys directly and must request encryption and decryption operations through ensemblewarden: Only ensembleporterd and, on follower nodes, ensemblefollowerporterd hold the entitlements that let a process take request key material from cb_jobhelper. Only ensemblewarden has the entitlement to fetch keys for cryptographic operations. This separation helps ensure that if a TIE process is compromised, it cannot access or supply key material. Each ensemble node uses a per-request process called ensemblewarden to manage encryption and decryption operations using the KVDEK. When ensemblewarden starts up, it retrieves the KVDEK from the local EnsemblePorter process (ensembleporterd on the leader or ensemblefollowerporterd on followers). As KV cache data is produced during inference, tie-inference asks ensemblewarden to encrypt the data using the KVDEK; ensemblewarden hands the ciphertext to ensembletransportd, which moves the encrypted blobs to the decode-phase ensemble. The underlying network path is described in Cross-node blob transport. In addition to the use of the KVDEK derived from user-provided key material, the prefill-decode handoff design takes additional steps to help ensure the cache data is accessible only to PCC nodes operating upon a specific request: The ensemble members exchange metadata over Attested TLS and validate each peer’s attestation against the policy that ensemble pairing uses. The prefill-phase ensemble generates a random cookie value, and the Transitive Trust passes it to the decode-phase ensemble. The decode-phase ensemble uses the cookie to identify the cache entry it fetches. To help maintain the confidentiality of cached data: Cached data has a short Time-to-Live (TTL) and is automatically cleaned up after expiration. The prefill-phase ensemble deletes each layer of the KV cache from memory once the decode-phase ensemble has read it successfully. Any layer that isn’t deleted this way expires under the TTL. Entitlements and sandboxing limit access to the key material used for encrypting the KV cache to only essential processes. note: EnsembleWarden KV cache reuse For multi-turn interactions handled by the Multi-Turn Agent, TIE can reuse the KV cache produced by one turn when handling the next turn of the same interaction. Recomputing the cache on every turn is expensive; for conversational and iterative workloads where each turn extends a common prefix, recomputation dominates the cost of inference. On the second and subsequent turns of an interaction, TIE tokenizes the prompt and MetalLM looks up the KV cache produced by the previous turn, continuing computation from the existing cache rather than recomputing the shared prefix. For each turn, the user’s device generates a fresh random 256-bit AES key. Alongside that turn’s request, the device transmits this new key together with the keys for any earlier turns of the same interaction whose cached state may still be reused. As tie-inference produces KV cache data, ensemblewarden seals each blob under the relevant turn’s key before handing the ciphertext to ensembletransportd, the per-node daemon that stores encrypted blobs and returns them on subsequent fetches — including cross-node fetches over ensembletransportd’s peer transport, described in Cross-node blob transport. Unencrypted cache data is held only inside tie-inference and inside ensemblewarden for the duration of an encryption or decryption operation; ensembletransportd itself does not observe unencrypted model state. This helps preserve PCC’s Stateless Computation property even though turn N+1 reuses KV cache entries derived from turn N. The encrypted blobs do persist on the node between turns, but PCC retains no KV cache encryption keys between turns: while no turn is being processed for an interaction, the encrypted blobs that remain in ensembletransportd are not decryptable by any process on the node without the device resupplying the relevant key. Continuing the interaction requires the device to resupply the keys for any prior cached state that is to be reused. Turn keys persist unencrypted only in per-request processes: cb_jobhelper decodes them from the request, and ensembleporterd, along with ensemblefollowerporterd on follower nodes, holds them for the request. CloudBoard starts ensembleporterd for each inference request and tears it down when that request’s job ends; on a follower node, ensembleportercontrollerd starts a fresh ensemblefollowerporterd for each request. A running porter takes one set of keys and offers no interface for replacing them, so it can’t serve a later request. ensemblewarden fetches the relevant key for each encryption or decryption operation rather than keeping a copy, and the long-lived daemons that span turns handle only ciphertext or sealed boxes they can’t open. Because each turn arrives as its own request, a turn’s keys leave the node when that turn’s work finishes rather than when the interaction does. When beginning a turn, the server reconciles the client-supplied metadata against the encrypted cache blobs it holds for the interaction: when MetalLM reuses a cached prefix, it compares the cached token sequence against the current prompt’s tokens and resumes computation from the first divergent token, recomputing only the divergent suffix. If cache reuse fails, the server falls back to full recomputation. Reuse can fail when the device omits a key for a referenced cached prefix, when follower-side validation rejects the cached state, or when a cross-node transport failure prevents the blob from being fetched. Cross-node blob transport ensembletransportd, a long-running daemon, performs the cross-node movement of encrypted KV cache blobs between PCC nodes. Two flows drive blobs through this path: Prefill-decode disaggregation, where ensembletransportd ferries blobs encrypted with the DEK-derived KVDEK between ensembles. KV cache reuse, where ensembletransportd is also the on-node store for blobs encrypted with per-turn keys supplied by the user’s device. In both cases, ensemblewarden performs the encryption before any blob reaches ensembletransportd. For cross-node fetches, ensembletransportd instances on different PCC nodes exchange request metadata over gRPC backed by Attested TLS. The daemon holds its own SEP-backed Attested TLS identity for this channel and validates each peer’s attestation against the policy that ensemble pairing uses. ensembletransportd periodically refreshes the attestation bundle through CloudAttestation so that long-lived peer connections remain bound to fresh attestations. Once metadata has been exchanged, ensembletransportd streams the blob bytes themselves over raw TCP. Because every blob handed to ensembletransportd is already encrypted by ensemblewarden, the raw-TCP data path does not weaken the confidentiality of blob contents. ensembletransportd holds published blobs in DRAM and may spill resident blobs to the ephemeral data volume under a TTL and disk-size cap. Blobs fetched from another node are not written to disk by the receiving node. Because PCC nodes run with Ephemeral Data Mode, any persisted blob is cryptographically inaccessible after a node reboot. Encrypted thinking traces Some PCC workflows invoke the inference engine in a mode that produces a thinking trace: an intermediate sequence of reasoning tokens the model generates before its visible response. The trace needs to be available on subsequent calls so the model can continue from where it left off. Because conversation state lives on the user’s device, the device round-trips this content to PCC on each turn. PCC returns these traces to the device in encrypted form. PCC Agent performs the encryption and decryption inside the PCC trust boundary, using key material delivered through a Codeless cryptexes. The protection narrowly targets model reasoning; prompts and visible response tokens continue to travel as cleartext within the Application protocol.