Google Cloud Node Hardening

A node reduces its attack surface, constrains what code can run, and measures everything that does.

Overview

A node boots Google’s attested Container-Optimized OS (aCOS), the image line that Google hardens for confidential workloads and that runs in Confidential Space. A node therefore runs a reduced package set on integrity-protected read-only file systems, takes no runtime updates, and exposes no SSH server. The host and guest operating systems, the hypervisor, and the guest firmware are all open source and Google Cloud Software Transparency.

Apple runs a node on a Bare-metal A4 instances, so the host operating system and the hypervisor sit inside PCC’s High-security inference and are not under the cloud operator’s control. Both are measured and attested alongside the Workload VM, and the measured stack reaches down to the firmware.

Apple builds the node’s runtime on aCOS to meet PCC’s requirement for a Hermetic environment, where every component trusted with user data is measured and attested. The aCOS image fixes its executable set at image-build time, and Apple extends the same constraint to the software it runs on the node.

A node’s software is split between Google’s The software on a node. The hardening in Google’s aCOS images and Intel TDX together isolate the Workload VM, which processes user data, from the host and everything else on the machine. Within the Megacontainer, Apple’s own hardening limits what a compromised component can run or reach: a sandbox per component, an execution allow list, a script-execution policy, and the separation of measured from unmeasured configuration.

Host and guest OS hardening

A node runs a host operating system, a hypervisor, and a guest operating system. PCC requires all three to meet the same goals:

Minimizing the attack surface by reducing components to those required by the workload
Hardening against both external exploitation and lateral movement within the node
Constraining execution to measured and attested code

Google distributes aCOS as a single disk image that holds everything beneath Apple’s workload container. That disk image carries the guest VM images for the Workload VM and the Protecting request keys VM, both on the same dm-verity-protected file system. The host and guest operating systems build on the same aCOS foundation and run in different isolation contexts, each with its own Multiple hardware roots of trust:

Host operating system: The operating system in that image, which runs directly on the hardware. It hosts the node’s confidential VMs using open source KVM and QEMU.
Guest operating system: The operating system inside the Workload VM, isolated from the host by Intel TDX memory encryption. Its UEFI guest firmware is the open source EDK II.

Both images share these properties:

Minimal: Each carries a reduced set of installed packages and kernel modules, which shrinks both the attack surface and the Hermetic environment.
Immutable and measured: dm-verity covers the root and OEM partitions, so a modified block fails to read rather than yield modified contents, and runtime updates are disabled, so an image’s own files cannot be modified in place. Each image boots from a Unified Kernel Image, which packs the kernel and its command line into one binary, so the command line and the dm-verity root hash are measured with the kernel rather than parsed separately by a boot loader. The kernel mounts the root file system directly.
Hardened kernel: The kernel is built with bounds checking, fortified memory and string routines, an allocator that masks freelist pointers with a random cookie, and indirect branch tracking from Intel Control-flow Enforcement Technology. Kernel crash dumps and unprivileged loading of eBPF programs are disabled, and the legacy kexec_load system call is compiled out. The kernel pins module and firmware loading to the verified root file system on the first such load and latches that setting for the rest of the boot, so nothing can relax it at runtime, and it requires a valid signature on every module it loads.
Write-xor-execute: Mounted file systems enforce write-xor-execute. Each image mounts its executable file systems read-only, and mounts the writable partition and every memory-backed file system noexec, with two exceptions in the Workload VM: the storage that holds the Megacontainer’s unpacked image layers, and the writable overlay root file systems the Megacontainer assembles for its components. The noexec option stops the kernel from running a file as a program or mapping it executable; an interpreter can still read a script from such a file system. Inside the Megacontainer, the script-execution policy governs those reads.
Ephemeral writable storage: Each image carries exactly one writable partition, which holds /var and /home, and everything else writable on a running node is either a memory-backed file system or a bind mount onto that partition. Each image encrypts that partition with dm-crypt in AES-256-GCM mode over dm-integrity, with a random initialization vector for every sector, and dm-integrity keeps each sector’s authentication tag in memory. A random-number generator seeded by the kernel produces the key at boot, and the key is never persisted, so the partition’s contents are unrecoverable once the machine restarts. A confidential VM’s partition sits on a virtual disk that the host holds in its own memory, outside the VM’s encrypted memory, so the host can read and write it. Authenticated encryption with the tags held inside the VM lets the VM detect a sector the host has modified or replayed; a confidentiality-only mode such as AES-XTS would not. The measured command line of each image sets cos.protected_stateful_partition=m, and the boot-time mount generator builds the device from that flag; the partition mounts only after that setup succeeds.
Fixed executable set: An aCOS image’s executables are fixed when the image is built, and the kernel’s Integrity Measurement Architecture holds the running system to that set: it appraises every file the kernel loads as a program, and every executable memory mapping, in enforcing mode, and returns EACCES for any file whose contents don’t match its recorded hash, or that lacks a recorded hash. Because both images mount their executable file systems read-only, and because a file written after boot lacks a recorded hash, nothing short of root can add to the set on a running node. A different mechanism covers the workload: its executables live on overlay file systems, which the Workload VM’s IMA policy exempts from appraisal, and an execution allow list enforces which of them may run.

Google builds the hypervisor to the same standard:

The build disables QEMU’s default device set and specifies each device it compiles back in: a PCIe chipset, the virtio block device that backs the VM image, VFIO passthrough for the GPUs and the Bare-metal A4 instances that carry node traffic, virtio networking, a vsock channel, and CXL memory and PCI expander bridges. Selecting the Q35 chipset also includes its parent PC platform, so the emulator provides that platform’s own hardware as well, among them an AHCI controller, an IDE bus, and a PS/2 controller. The host starts each VM with SATA off and attaches only the virtio block, virtio network, vsock, and passthrough devices.
The build enables stack protectors on every function; fortified memory routines; a position-independent executable; traps on signed integer overflow; and immediate symbol binding with a read-only global offset table.
The hypervisor runs from a minimal container image on the host, built without QEMU’s guest agent. The libvirt build in that image disables the network driver and every virtualization driver other than QEMU.
At runtime, the hypervisor starts with no display and no default devices, reachable only through the control socket libvirt uses to manage the domain, with a guest serial console only on debug images, and with the live VM image in a tmpfs. Each VM’s disk is a copy-on-write overlay in that tmpfs, backed by the read-only image the host image carries, so a VM’s writes accumulate in host memory and are discarded when the VM exits. The host compares the overlay’s size against its available memory before launching, and refuses to start a VM whose overlay it cannot hold.

Workload hardening

At boot, the host launches the node’s confidential VMs, and inside the Workload VM, The software on a node downloads, measures, and launches a single container image named in the instance metadata.

Before any container code executes, the launcher extends the image reference, manifest digest, configuration digest, container arguments, and environment variables into the VM’s measurement registers and appends them to the event log. The container image also carries policy labels that restrict which launch parameters an operator may override; because these labels are part of the measured image, an operator cannot expand the permitted overrides without changing the image hash and invalidating the release.

PCC gates authorization on remote attestation rather than on local code signatures. A client or proxy node Node validation, verifies that the node’s measurements, including the container digest, match a release recorded in the public Release Transparency, and only then releases the keys needed to decrypt a request. The node makes no part of that decision itself.

Beneath the container, two kernel mechanisms protect the Workload VM’s own operating system files. dm-verity checks each block against the volume’s hash tree as the kernel reads it, and kernel integrity appraisal checks each file against its recorded hash before the kernel executes it; the kernel rejects a block or file that doesn’t match. The Workload VM’s IMA policy exempts overlay file systems and the writable partition that holds the container image so the container can unpack; attestation then authorizes the container from outside the node, and the execution allow list constrains what runs inside it.

Workload Megacontainer

That image is the Megacontainer. Packaging the runtime as a Megacontainer brings all of Apple’s software on a node under a single measurement, while the components inside remain independently built and isolated from one another.

[Image]

Each of Apple’s The software on a node is compiled into a separate OCI image with its own userland, so it carries its own dependencies and builds independently of the others. A node doesn’t download additional code at runtime, so every executable the node starts comes from these images. Those images are assembled into the Megacontainer, where systemd manages the individual jobs, and a container runner (runc) launches each bundled OCI image as a systemd unit. Using standard systemd and runc primitives provides established namespace isolation and process life cycles without requiring custom supervisor daemons.

Each component runs in its own process, mount, and IPC namespaces, and the build specifies, per component, what else that component may reach: its user and group, its capabilities, the file systems it can see, its network namespace, its device access, and its resource ceilings. A component’s capabilities, mounts, device access, namespaces, and resource ceilings default to the restrictive case, so widening any of them requires an explicit declaration, which makes each exception reviewable. A component runs as root unless its entry declares a user, so the manifest names a distinct unprivileged user for every component that doesn’t need root. Components reach one another only through channels granted the same way. systemd creates and holds each component’s listening socket and starts the unit when a connection arrives; a component’s mount namespace carries a bind mount only for the sockets that component is meant to reach.

Each request runs as a systemd template unit in Per-request isolation that is destroyed when the request ends. systemd forces that teardown to complete, so a per-request context is reclaimed even when the request’s processes do not exit on their own.

The image build inventories the executables in the Megacontainer: it scans the base image and each component image and records a hash for every binary it finds, producing a digest catalog.

Execution enforcement

Inside the Workload VM, each component’s root file system is a writable overlay that the Megacontainer assembles from the component’s image layers, with those layers and each invocation’s writable upper layer on memory-backed file systems. The VM’s IMA policy exempts overlay file systems from appraisal, so dm-verity and IMA hold the VM’s own operating system to its image but leave these root file systems unchecked. Without a further check, a compromised component that can write under its own root, or overwrite a binary in place, could run code that no release records. The execution allow list is designed to hold each container to the executables its build recorded.

The execution allow list and the script-execution policy apply to the Megacontainer: the processes in its PID namespace and the component containers within it. A process on the host, or in the Workload VM outside the Megacontainer, falls under neither policy. Those operating systems rely instead on their fixed executable set, and on hardened images exposing no SSH server.

The privileged lsecuritymonitorlited daemon, which reports the node’s Security event monitoring, enforces the digest catalog. At runtime, the daemon loads the catalog into the kernel as an execution allow list and attaches eBPF programs to the kernel’s Linux Security Module (LSM) hooks through the BPF LSM; those programs check each execution against the list. PCC compiles the decision to enforce into the daemon’s binary rather than reading it from configuration, so unmeasured and dynamic configuration cannot disable or relax the policy.

Each component image includes an allow list manifest under /mc/eh-allowlists/ that maps executable paths to SHA-256 digests. Because these manifests ship inside the measured container image, the attestation covers the allow list a node enforces. At container startup, the daemon reads /mc/eh-allowlists/base.json from within the container’s mount namespace, so one configured location yields a different manifest per container, and resolves each entry to its inode and device number there before loading the identifiers and expected digests into kernel maps. The daemon enrolls the container before loading the manifest and enables enforcement only after the load completes. Each container therefore has a startup window during which its allow list isn’t yet in effect. Once enforcement is active, the kernel runs a file in that container only if the container’s manifest lists it, with one exception: in the Megacontainer’s base scope, the daemon admits any file whose contents match the runc binary the base manifest lists, wherever that file resides. The daemon applies a catalog of the Megacontainer’s own base root file system through the same mechanism.

Execution checks run at the kernel’s bprm_check_security LSM hook, which fires for every file a process asks the kernel to load as a program. The eBPF program looks up the file’s inode and device number in the container’s allow list, directs the kernel’s Integrity Measurement Architecture to compute the file’s SHA-256 digest, and compares the result to the recorded value. On a miss or a mismatch, execution fails with EPERM before any instruction of the file runs, and the daemon reports the denial as a security event. Because the check keys on the file’s identity and content, a file receives the same verdict under any name, link, or mount path.

The check fails open when:

The program cannot resolve the file’s inode or path.
The program cannot attribute the execution to a container the daemon tracks.
The container’s allow list is not yet in effect.
The kernel’s hash computation itself fails, as on an I/O or memory-allocation error. The check then matches the file’s inode and device identity alone and authorizes the file without verifying its contents.

The Megacontainer build writes a manifest for every component and fails if a manifest exceeds the entry limit the daemon accepts, and each component’s container configuration mounts its manifest read-only at the path the daemon reads. A component therefore falls back to the daemon’s directory scan, which admits the executables it finds in the container’s binary directories, only if its manifest can’t be read or yields no entries.

The allow list doesn’t check a library mapped into a running process, even one a manifest lists. Executable mappings are subject instead to the Workload VM’s Integrity Measurement Architecture policy, and its coverage varies by file system:

Appraised file systems: The Workload VM appraises every executable mapping on a file system the policy covers, including the memory-backed file systems a process can create for itself. An ELF written into an anonymous memory file (memfd) or onto /dev/shm therefore fails to map executable, whether a process reaches it through dlopen, through LD_PRELOAD, or by invoking the dynamic loader against the descriptor.
Container root file systems: The policy exempts overlay file systems, and a component’s root is an overlay, so a mapping from a component’s root isn’t appraised. A component that can write a file under its own root can map that file executable, and it can map anonymous executable memory with no file at all. Namespace, user, and capability isolation bounds the code a component can introduce.

Script-execution policy

Interpreters complicate execution enforcement because the execution allow list checks a file the kernel loads as a program, and an interpreter opens its script as data. When an interpreter runs a script, the kernel sees only the interpreter binary. An attacker with container write access, whether through mounted volumes, compromised dependencies, or temporary directories, can therefore execute arbitrary code by passing an unverified script to an authorized interpreter.

Because PCC inference workloads depend on Python, we cannot remove interpreters from the node. PCC instead extends execution enforcement to the files a recognized interpreter loads as code.

The build records Python source files and bytecode as script entries in each component’s manifest, and lsecuritymonitorlited loads them into a script allow list alongside the execution allow list. The daemon registers the names of the Python interpreters it recognizes, and attached eBPF programs gate both how a registered interpreter is launched and what it opens once it is running.

The daemon blocks three invocation patterns that lack corresponding recorded files: inline scripts (-c), code arriving on standard input, and bare interactive interpreters. eBPF programs attached to the kernel’s exec tracepoint inspect process arguments and file descriptors; if an interpreter matches these prohibited patterns, the program sends the new process SIGKILL before it runs its first user-space instruction.

To reach the code an interpreter reads at runtime, eBPF programs also intercept file opens. Whenever a registered interpreter opens a .py or .pyc file, including imported modules and the standard library, the eBPF program verifies the file’s inode, device number, and SHA-256 digest against the script allow list. Files absent from the allow list fail to open with EPERM. If the kernel cannot compute a digest, the check falls back to matching inode and device numbers. An extensionless script doesn’t reach that test by name, so the daemon resolves the interpreter’s command-line argument and PYTHONSTARTUP path against the process’s working directory. The allow list records no extensionless file, so a file at either path fails the allow list check, and the kernel terminates the process.

A component that requires inline execution can declare reduced script blocking in its manifest, measured with the image like every other declaration. This declaration downgrades only the inline code and standard-input checks to audit mode for that container. The daemon continues to enforce allow list matching on script files and terminates bare interactive interpreters.

Limitations

The policy’s coverage has these limits:

It applies only to Python. A component that bundles another interpreter runs that interpreter’s scripts unchecked.
It applies only inside the Megacontainer. A Python interpreter on the host or in the Workload VM’s own root file system is unchecked.
It requires the interpreter to run as its own process. A program that links CPython as a shared library, as a node’s model-loading component does, runs Python code without executing an interpreter binary.
As with the execution allow list, a container’s scripts run unchecked until its script allow list loads.
The daemon tracks interpreter processes in a fixed-size table that evicts the least recently used entry once it fills, so an evicted process’s later script opens go unchecked and the daemon records no denial.
Standard-input checks detect only pipes and regular files, so an interpreter reading code from another descriptor type, such as a socket, is unchecked.
An unlisted .py or .pyc file fails to open, but the kernel catches an unlisted extensionless script only after open succeeds, so the interpreter briefly holds its contents before the kernel terminates the process.
Checking a file at open leaves a race with the interpreter’s later read. Eliminating the race requires the interpreter to ask the kernel whether the file it opened may run as code, using the AT_EXECVE_CHECK flag added in Linux 6.14. The node doesn’t have that today: its guest kernel is older than Linux 6.14, and the bundled CPython doesn’t make the request.

Measured, unmeasured, and dynamic configuration

A node distinguishes configuration that its attestation covers from configuration the control plane supplies at or after boot. Because Confidential Space measures the container image, its arguments, and its environment variables, and nothing else the control plane supplies, security-relevant configuration must be measured through those channels rather than delivered out of band.

Configuration on a node falls into three kinds, distinguished by whether it is covered by the attestation and, for the two kinds delivered by the control plane, whether it arrives at boot or after boot:

Measured configuration: Built into the container image and covered by the attestation. Any value that is security- or privacy-relevant must be measured configuration: whether log-audit-list enforcement is on, the digests a node checks its startup assets against, the recycle interval the inference stack applies.
Unmeasured configuration: Delivered by the control plane at boot and not covered by the attestation. It carries values that vary by region, deployment environment, or variant, such as service-discovery URLs and expected peer identifiers.
Dynamic configuration: Delivered by the control plane after boot and not covered by the attestation. It exists for values that must change without a node reboot.

PCC’s threat model treats unmeasured and dynamic configuration as attacker-controlled: these values must not be able to change the security or privacy properties of the node, and the components that consume them are expected to validate them accordingly.

[Image]

The container image includes measured configuration for multiple variants. The launch-time environment variable PCC_VARIANT, which Confidential Space measures into the attestation, selects the configuration set. A variant’s measured configuration is cleartext JSON, one file per component, so the whole measured-configuration set is enumerable from the published image.

A component reads measured and unmeasured configuration through separate readers, and where one has to merge both into a single namespace, it applies its measured configuration last, so a measured value wins a collision.

Startup assets

A node’s release covers its boot chain, firmware, and container image, but at startup a node can also download assets, such as model weights and prompt packages. Without coverage of these assets, a client could verify the code a node runs but learn nothing about which weights or prompts it loaded. To close that gap, the build records the digests of these startup assets as measured configuration inside the container image, so the container image measurement in the attestation covers them and the release recorded in the Release Transparency covers them in turn. Each asset’s measured configuration records the digest the node expects. The node checks each asset against that digest as it downloads the asset, and refuses to start if they don’t match. The configuration also records each asset’s source, but the node’s check relies on the digest alone.

Because the digests travel in the container image, an update to these assets produces a new container image measurement and therefore a new log entry, allowing researchers to see from the log that a node’s asset set changed.

Startup assets are measured but not published, so PCC’s threat model treats their contents as attacker-controlled. To keep a malicious asset from executing code, the runtime accepts only restricted data formats:

Model weights arrive in SafeTensors format, consisting of a JSON header defining tensor shapes, data types, and byte offsets, followed by raw binary data. The loader copies these byte buffers directly into memory without using executable deserializers such as Python pickle.
Model architecture configurations arrive as structured JSON. The runtime deserializes the configuration into typed structures whose fields accept only a fixed set of values, so a configuration can only select operators already compiled into the measured container image.

Both formats carry data alone: neither can encode control flow, and the operators a configuration selects are already compiled into the measured container image.

Node network isolation

A node runs inside a Virtual Private Cloud (VPC) whose firewall rules govern both directions of its traffic: they admit inbound connections only from the PCC routing layer, and permit outbound connections only to an allow list of Apple service endpoints the node needs for request routing, service discovery, authentication, and telemetry. The surrounding network enforces those rules rather than the node, so a compromised component cannot widen its network access. However, because attestation does not cover a VPC’s network configuration, PCC’s verifiable guarantees rest on the node’s own namespace isolation and cryptographic controls.

Inside the node, the boundary narrows again, because a component reaches the network only if its unit asks to. Each unit declares the namespaces it shares with the VM. A unit that names the network namespace reaches the network through the VM’s interfaces, bounded by the allow list; a unit that does not gets an empty namespace of its own, with no interfaces and no route to the network at all, and reaches its peers only over the Unix sockets the image grants it. Roughly a third of the components a node runs are in the second group, including the per-request job helper. These declarations are built into the container image rather than supplied at runtime, so which components can reach the network falls under the same measurement as the code.

The host sits outside the workload’s data path. Both of the machine’s physical network interfaces are passed through to the Workload VM, so once that VM starts, it holds the machine’s only connection to the outside network, and the host cannot observe or interpose on request traffic. The host does keep a private bridge joining the Workload VM to the Key Protection Service VM; the daemons on either side seal the key material they exchange under HPKE, so the host handles only sealed payloads.