Management and Observability

PCC nodes support privacy-aware management, observation, and debug capabilities.

Overview

Like any large-scale service, PCC requires management, introspection, and debugging capabilities. These requirements conflict with our security model, which establishes defensive barriers against introspection and debugging regardless of who performs them. We must balance effective administration of the service with our security goals to keep user data and identity confidential.

To strike that balance, the PCC design enumerates each category of telemetry Apple’s software produces and each point at which that data can leave the node. Each of those export points filters what it forwards, helping ensure that routine systems-management operations don’t create potential user-data exposures.

There are broadly three kinds of telemetry within the trust boundary:

Logs: Text events, structured or unstructured.
Metrics: Time-series data reported near real time for monitoring and alerting.
Distributed traces: Structured events that track a request as it flows through the system.

We designed PCC to ensure that two properties hold across every export point:

The filtering rules that decide what may leave a node are carried in the node’s published, measured software, so independent parties can inspect and verify them; the rules are covered by PCC’s Verifiable Transparency.
A specialist team audits each change to the metrics, log lines, and traces in those rules before deployment.

PCC scopes the granularity of those rules to a component’s exposure to user data. Where a component’s address space can hold user data, the rules admit individual log lines and individual metrics, each reviewed on its own. Where a component never receives user data, as with the daemons that manage attestation or that download assets, the rules admit that component’s telemetry as a unit.

This audit-and-transparency model is the same on both Apple silicon and Google Cloud, though the mechanisms that enforce it differ.

Management and observability on Apple silicon

An Apple silicon PCC node runs a derivative of the Apple operating systems and reuses their privacy-conscious telemetry infrastructure, constrained further for PCC. For completeness, here are some management tools and techniques that we replaced or intentionally left out:

Interactive shell access: Interactive system access for site reliability engineers presents significant privacy issues, because of the large interface surface from which user data might be exposed.
Sysdiagnose: Sysdiagnose is disabled when a PCC node enters Restricted Execution Mode. Instead, PCC collects only specific and necessary data. This targeted approach differs from iOS and macOS, which rely on sysdiagnose archives to investigate issues.
OS analytics: PCC disables the operating system’s opt-in reporting of analytics, diagnostics, and usage information, in favor of purpose-built mechanisms that can apply PCC-specific privacy policies.
Execution traces: Detailed execution-trace recording features, such as those that power the Instruments Time Profiler, provide high-resolution execution data that poses a risk of exposing user data through side channels. A node ships none of the tools or daemons that drive those features.

A PCC node runs with Ephemeral Data Mode, which helps prevent user data from surviving a system reboot, so no one can reconfigure the system to reenable any of these tools while retaining user data.

Metrics

The funnel point for all metrics reporting on an Apple silicon node is CloudMetrics, an implementation of the swift-metrics API back end. Software in the Private Cloud Extensions uses this implementation to log counters, gauges, and histograms. Observations reach the cloudmetricsd daemon over XPC from the CloudMetrics framework, or through the os_log event stream from the os_metric APIs, and the daemon reduces them with local temporal aggregations. The aggregates are exported at a regular interval to an Apple metrics service through the OpenTelemetry Protocol, an open observability standard.

Before cloudmetricsd exports anything, it checks each observation against a restrictive allow list of exportable metrics. The check runs in the daemon rather than in the emitting process, so a compromised component cannot bypass it. The daemon also controls the frequency of metric exports and drops metrics based on their name, allowed dimensions, or range of valid values. The PrivateCloudSupport cryptex includes these configurations, so they are subject to our verifiable transparency guarantees.

CloudBoard and The Inference Engine use CloudMetrics to record application-level metrics. The system also includes devicemetricsd, which collects system-level metrics — including performance and networking events — and routes them to CloudMetrics.

In-node aggregation

The behavior of some PCC mechanisms depends on the content of a request. Collecting metrics about these mechanisms requires care, because both the value of a metric and the timing of its export can carry information about that content beyond the trust boundary. For example, Recitation prevention blocks a result based on the content of the model’s output. A counter that increments each time recitation prevention blocks a result does not reveal a particular request, yet an observer who correlates the timing of its export with the requests a node is actively processing could infer that a particular request was blocked.

PCC’s primary strategy for such situations is data minimization: collecting only the information essential to operating the service. To limit what these metrics can reveal about a request, we designed PCC to support in-node aggregation.

For metrics configured for in-node aggregation, cloudmetricsd retains observations within the trust boundary and exports them only after a configured minimum time window has elapsed and a per-bucket minimum number of events has accumulated since the last export. Each bucket carries a persistent threshold noise: an independent Laplace value that adjusts the bucket’s publish threshold. This value is held constant across window evaluations and is redrawn only after the metric is published. At each evaluation, the daemon adds a second, freshly drawn Laplace value to the bucket’s count and compares the noisy count against the noisy threshold; if any bucket falls short, the entire histogram is held until the next window.

We designed this mechanism to resist attacks even from privileged network positions:

Holding the threshold noise fixed across evaluations, rather than redrawing it each time, denies an adversary who watches publish timing the ability to repeatedly re-roll the threshold and estimate how close a bucket’s true count sits to it.
Requiring that every bucket meet its threshold helps defend against an adversary who attempts to force publication by inflating events in one bucket alone, because the other buckets would also need to reach their typically much larger thresholds.

When the thresholds are met, the daemon adds a third independent Laplace value to each bucket’s count before export, with a scale sized to be large relative to a single event’s contribution, but small relative to fleet-scale aggregates; this is designed to mask any single event’s contribution to the published value while preserving utility in aggregate.

In-node aggregation is enabled on a per-metric basis. The enablement and aggregation parameters are part of the same allow list configuration as the metric-filtering rules and are subject to the same verifiable transparency guarantees. The aggregation parameters are in the configuration files of the PrivateCloudSupport cryptex, which researchers can Downloading available releases.

Log filtering

An Apple silicon node generates logs using the privacy-conscious os_log(3) API, just like iOS and macOS. The splunkloggingd daemon provides an export service for this log data that conforms to PCC’s privacy commitments and is tightly integrated with the os_log subsystem. Because each format string used with os_log results in a distinct log event, splunkloggingd can impose filtering on a per-log-line basis.

splunkloggingd exports a message only if the log line is permitted by its filtering rules. Like the permitted metrics configuration, the allow list that defines the permitted messages is included in the PrivateCloudSupport cryptex and subject to independent inspection. While os_log is used throughout the PCC software stack, only a select set of those messages are exported from the node.

The customer security policy requires log persistence to be off, so a node also keeps its log data in memory only until splunkloggingd either forwards a line or drops it.

An entry in the allow list is composed of the Mach-O image that calls os_log and the format string. A strict review process audits existing and new os_log calls in the PCC stack to determine which log lines are eligible for inclusion in this allow list. Because os_log format strings capture the type information of the variable arguments, privacy review also considers the data types included within a log line.

splunkloggingd matches each log event against the allow list on the Mach-O image that emitted it, and drops an event from an unlisted image. Each event contains both the fully composed log line with interpolated arguments and the original format string from the call site; comparing that format string against the allow list lets the daemon forward only pre-approved logs.

Crash reporting

Triaging and analyzing user-space process crashes is a critical engineering task for all Apple platforms, and the PCC node operating system uses the same crash-report generation infrastructure as macOS and iOS. This infrastructure is already built to be privacy-preserving, and for PCC the system considers two types of data included in crash reports:

Intrinsically safe: Static data or system identifiers such as OS version or process ID. Intrinsically safe attributes can’t accidentally reveal sensitive data, because they are determined early at boot or in a context without any user data present.
Process-state dependent: Data determined by inspecting the state of the crashing process, such as stack backtraces and register state. While already collected with strict user-privacy principles in mind, process-state-dependent attributes carry some risk of accidentally revealing user data.

PCC’s privacy mitigations focus on reducing the amount of process-state-dependent data that can leave the node. To handle this data, the crash-report infrastructure defines two redaction behaviors, partial and full. splunkloggingd applies one of them to every report it forwards, and records which one in the report’s redactionLevel field. They limit the data as follows:

Crash log attribute	Partial redaction	Full redaction	Notes
Register state	Redacted	Redacted
Unresolvable stack frames	Redacted	Redacted	Bogus stack frames might indicate the stack is overwritten with runtime data.
Exception information		Redacted
Thread backtraces		Trimmed	Under full redaction, the report keeps the crashing thread’s first 32 frames and drops all other threads’ backtraces. Under partial redaction, thread backtraces are kept in full.

A maximum of three partially redacted logs is reported from each node per hour. Partially redacted reports are emitted randomly for 20 percent of crashes up to this limit.

In addition to crash logs, splunkloggingd also forwards panic logs. If a node panics, DumpPanic writes a report on the next boot, and splunkloggingd forwards that report to Splunk. splunkloggingd applies no redaction of its own here; it prepares the report through the same shared DumpPanic interface that iOS and macOS submissions use. Because the fatal nature of a system panic naturally limits how often one occurs, PCC applies no additional rate limiting to panic reports.

Remote diagnostics and monitoring

An Apple silicon PCC node has neither remote access nor a local shell, so traditional methods of gathering administrative health data are not available. Instead, we built specific interfaces to collect health and diagnostic information through the CloudRemoteDiagnostics project and its cloudremotediagd daemon. This daemon uses system interfaces to directly collect a limited set of statistics and doesn’t spawn helper processes for this purpose. It makes these interfaces available only to its paired baseboard management controller (BMC) over the in-chassis network.

We designed each function of CloudRemoteDiagnostics to minimize the risk of user-data exposure. For example, the daemon exposes functionality to capture stack traces of potentially misbehaving processes. As with crash logs, these backtraces pose only a minor risk of leaking user data; but if applied at sufficient frequency, this functionality might expose the system to a side-channel attack. To mitigate this risk, the daemon imposes a rate limit on this functionality, limiting the effective bandwidth of any potential side channel.

The capabilities of this daemon include:

Basic connectivity tools: Provides ping and traceroute to a remote endpoint.
Network data capture: Produces a packet capture of traffic on a named interface and port. Every capture applies a fixed filter that drops any packet addressed to or from the node’s loopback addresses, so the file records only traffic that enters or leaves the node.
Stackshot: Captures a backtrace of all processes on the system, limited to one stackshot per hour.
CPU and memory usage: Collects statistics on which processes are using the most CPU, memory, or threads.

Service health monitoring

The systemservicemonitord daemon provides a centralized mechanism for system services to report their state and for management services to query that state. Monitored services establish XPC connections to the daemon and report their state — presented, ready, or failed — as they progress through initialization. Management services can discover which services have checked in, fetch their current state, and receive real-time notifications when a service’s state changes or becomes unavailable.

The daemon exposes a RemoteXPC endpoint that allows the BMC to query service health within the chassis. The daemon requires management clients to hold the com.apple.ssm.client.allow-request entitlement. It also requires monitored services to hold the com.apple.ssm.monitored-service-identifier entitlement, which provides the identifier used for reporting. systemservicemonitord does not introspect the services it monitors and does not handle user data.

Security events

An Apple silicon PCC node captures a limited number of security events through the securitymonitorlited daemon, which is a client of the EndpointSecurity framework and other system metadata sources. This daemon records the following security event types:

Process life cycle: Process exec, process exit, and process signal.
File events: File open and similar events for specific files and patterns.
IOKit usage: IOKit open.
SSH: SSH login and logout. SSH is disabled on production PCC nodes, so while these events are included for completeness, they typically don’t fire in normal operation.
Network: Connection opens through the private NetworkStatistics framework, the framework that powers nettop and Activity Monitor.
System log resources: Selected log artifacts, including policy enforcement for Restricted Execution Mode, sandbox violations, and system DNS queries.
Alerts: The securitymonitorlited daemon, while observing the events above, uses on-host detection logic to generate alert events containing the metadata that triggered the alert.

These events are sent directly from the securitymonitorlited daemon to a remote event-collection service, where Apple security teams analyze the data in aggregate to identify indicators of compromise. Based on available system resources, event-loss telemetry may be sent in place of full events.

The set of exportable events is enforced in the PCCFrontend implementation, which defines the behavior of the securitymonitorlited daemon. This daemon is delivered with the PrivateCloudSupport cryptex and is subject to our verifiable transparency guarantees. The event-capturing architecture prioritizes user privacy and deliberately doesn’t attempt to introspect or export data from within processes, instead preferring to capture metadata for the events.

Network firewall

When a PCC node boots, darwin-init applies a pre-specified network policy that allows the node to obtain an IP address, download the necessary cryptexes, and personalize them. The PrivateCloudSupport cryptex contains the node’s firewall agent: denaliSE. This agent controls traffic ingress and egress between the PCC node and the rest of the Apple data-center network and supports individual, stateful packet inspection with enforcement at layers 2, 3, and 4.

Once active, denaliSE communicates with the Denali control plane over mTLS, using a client certificate managed by the SEP. The control plane verifies the client’s identity and then computes a unique view of the Apple data-center network security policy for access patterns that are specifically applicable to that node. This security policy permits network access only to and from the minimal data-center services required to operate PCC. denaliSE downloads this computed policy and enforces it on every packet crossing the node’s uplink to the data-center network. denaliSE does not filter the in-chassis link to the BMC; access over that link is limited by entitlement checks on the diagnostic and service-health interfaces described above.

denaliSE regularly publishes metrics about the health of the firewall and its enforcement actions for consumption by data-center operators. The BMC also monitors the health of each PCC node’s denaliSE agent to help ensure that it is operating normally.

Management and observability on Google Cloud

A Google Cloud node runs a Linux-based stack rather than an Apple operating system, so it cannot reuse os_log, CloudMetrics, or the Apple crash-reporting infrastructure. It preserves the same privacy model — audited allow lists carried in the node’s measured software, enforced at every export point — through a telemetry pipeline built from off-the-shelf, industry-standard components. That pipeline handles three telemetry types: logs, metrics, and distributed traces. A node does not currently collect crash reports.

The privacy problem is unchanged from Apple silicon: a node must emit enough data to be operated and debugged without any of that data leaking user information. The additional difficulty on Google Cloud is that the node runs software we don’t fully control — system components such as systemd, and inference components written in languages other than Swift — so the pipeline can’t rely on a single Apple logging API to separate the fixed structure of an event from its variable, potentially sensitive contents.

Management tooling the images omit

A Google Cloud node leaves the standard Linux and cloud-provider management tooling out of its images rather than shipping it disabled:

Interactive access: OpenSSH is not installed on the host or on either guest image, so the node exposes no remote administration path.
Provider control-plane agents: google_guest_agent and google_osconfig_agent are not installed, so the cloud provider’s standard channels into the node have no agent to answer them.
Interpreted startup configuration: Fixed startup scripts replace cloud-init, removing a boot-time path that executes instructions fetched from elsewhere.
Diagnostic utilities: toolbox and node-problem-detector are absent, removing binaries an attacker could use to reach the host.
In-place updates: The images do not include update-engine, and each image mounts its root and OEM file systems read-only under dm-verity for the life of the boot, so an image changes only by booting a new measured one.

For the properties relaxed by the debug image, see Appendix: Google Cloud Debug and Hardened Images.

The telemetry pipeline

All logs, metrics, and traces from the workload components on a Google Cloud node converge on a single collector that runs inside the Workload Megacontainer. The collector is built from Fluent Bit, an industry-standard observability component. Workloads emit logs, metrics, and traces over the OpenTelemetry Protocol (OTLP) to the collector using a Unix domain socket that is mounted read-only into each subcontainer. Each component’s OTLP endpoint is hard-coded to that socket, so every component’s telemetry reaches that collector, whatever the node’s configuration says. The collector applies the audit filtering before anything leaves the node.

The collector uses off-the-shelf plugins where possible and adds custom plugins for the privacy-preserving requirements that are specific to PCC:

OTLP input plugins receive logs, metrics, and traces from workloads.
Additional plugins receive node-level metrics, Fluent Bit’s internal metrics, and the kernel, audit, and driver messages in the node’s journal.
Custom filtering plugins apply the log audit list, the metrics audit list, and the tracing audit list.
Off-the-shelf plugins add node-wide metadata to logs and metrics.

The audit lists, and the filtering rules that apply them, are part of the node’s Measured, unmeasured, and dynamic configuration: they are delivered in the attested container image, and the control plane cannot override them. The filtering the collector applies is therefore covered by verifiable transparency, just as on Apple silicon. Configuration that the control plane supplies at or after boot, such as how much log volume a node emits or where the collector sends it, is unmeasured, and PCC’s threat model treats it as attacker-controlled. The measured audit lists bound what the stream can contain, whatever that configuration says.

Beneath the Megacontainer, The software on a node runs collectors of its own for the platform’s operational messages. The host’s collector writes to the machine’s serial console and has no network output. The Workload VM guest’s collector reads entries from three units in the guest journal: the container launcher, the Appendix: Google Cloud Workload Services Daemon, and the network manager. It also reads kernel messages and the Protecting request keys VM’s journal of its key-management and attestation services, and forwards all those entries to Google Cloud Logging. These paths carry the platform’s messages about its own operation, and Apple’s audit lists do not govern them. Apple’s components reach the guest journal only through a Appendix: Google Cloud Debug and Hardened Images facility that redirects the container’s output streams into it.

Metrics

Metrics are the most tractable telemetry type for auditing, because they are inherently structured: a single time series is a set of flat attributes with one or more numerical values. Those attributes can nonetheless leak sensitive data in the same way a log line can, so the collector filters every metric against a metrics audit list, adds node-wide metadata, and drops any label that the audit list does not expect for that metric. For a component without access to user data, the audit list admits all of that component’s metrics and labels.

PCC authors the metrics audit list from two directions. For first-party components, we annotate the metrics in source so an audit list can be produced from the code. For third-party components that expose metrics as Prometheus endpoints, such as the inference engine, we rely on public documentation and qualification testing to identify the metrics they produce. A collector inside the inference runtime scrapes those endpoints and forwards the metrics to the node collector over OTLP, so the same audit list governs them. The node collector gathers host and per-process metrics itself, reading them from the kernel rather than from an endpoint.

Log filtering

Apple’s components in the Megacontainer emit structured logs over OTLP, where the fixed message text is separated from the variable, potentially sensitive fields. This mirrors the Apple silicon model, in which an os_log format string separates the fixed template from its interpolated arguments; the collector matches each log against the audit list on the same basis and forwards only pre-approved lines. For a component without access to user data, the audit list admits all of that component’s logs. As with Apple silicon, the audit list is carried in the node’s measured software.

Distributed traces

Applications on a Google Cloud node emit native OTLP traces that they send to the collector. The collector’s audit plugin supports matching spans and span events against the audit list, ensuring that each matches a pre-approved entry in the list. Each entry also enumerates the set of attributes that the auditors approved for the corresponding span or span event. The plugin redacts any attribute that the audit list doesn’t include for that component.

Remote diagnostics and monitoring

CPAgent, the control-plane agent inside the measured container, serves the node’s management listener. It reports node health, returns the node’s attestation for the control plane to record, and listens for drain requests.

cloudboardd reports the node’s own status to CPAgent, which answers the control plane’s health checks. The The inference pipeline registers the readiness of the AFM Runtime and the GPUs with cloudboardd, so those signals reach the control plane along the same path. Everything these components exchange is operational state: readiness, registered batch sizes, and component liveness.

Security event monitoring

A Google Cloud node captures a limited set of security events through the lsecuritymonitorlited daemon, the Linux counterpart of securitymonitorlited. Linux has no EndpointSecurity framework, so the daemon observes the system through eBPF programs that it attaches to kernel tracepoints, Linux Security Module hooks, and kernel functions. The daemon records the following security event types:

Process launches: The executable path, arguments, and process ancestry.
Network: Connections opened and accepted, and DNS queries.
Device access: Device nodes opened.
SSH: Login and logout events. Production images include no SSH server, so these events don’t fire in normal operation.
Blocked execution: Denials from the node’s Execution enforcement, carrying the denied file’s path and digest, and the process that attempted the execution.
Liveness: A heartbeat at a fixed interval and periodic snapshots of the process inventory, so that an absence of events is itself a detectable signal.

The daemon sends these events directly to the same event-collection service that Apple silicon nodes use, where Apple security teams analyze the data in aggregate to identify indicators of compromise. The set of exportable events is a fixed list of event schemas compiled into the daemon. Before export, the daemon maps each observation onto one of those schemas and drops anything that doesn’t match. The set can therefore widen only through a change to the daemon itself, which ships in the measured container image and is covered by our verifiable transparency guarantees. The events describe what a process did. Because a process’s arguments and the hostnames it resolves are among the recorded fields, PCC workloads keep request-derived text out of both. The daemon bounds its own event volume with flood protection, and it counts the events it drops rather than losing them silently.

An attacker who cannot run unauthorized code might instead try to stop the daemon itself. Detection rests on the heartbeat: the collection service expects one from the daemon at a fixed interval, and a missing heartbeat signals that the daemon stopped, whatever stopped it.

Virtual Private Cloud firewall

A Google Cloud node’s ingress and egress are bounded by Node network isolation the surrounding network enforces rather than the node, so a compromised component cannot widen its own access. Inside the node, each component runs in an empty network namespace of its own, and reaches the network only if its unit declares the VM’s network namespace.