Google Cloud Research Interactions

Issue requests to a research node, verify its attestation, and inspect what it does with a request.

Overview

Every procedure here assumes a tunnel to the instance and, where noted, a shell in its Workload VM; Google Cloud Research Setup covers both.

Issue inference requests

You can reach the instance’s workload port, which carries application traffic, through the tunnel. pcc-curl on your host validates the node’s attestation bundle, submits the request, and streams the response back.

The simplest request needs no payload file, because --user-prompt builds one for you. Point the client at the local end of the data-path tunnel:

% pcc-curl cbcurl \
    --hostname 127.0.0.1 --port 4442 \
    --user-prompt "What model are you?"

cbcurl is the default subcommand, so you can leave it out. Omitting --system-prompt runs the request against the model’s own default system instruction; pass it to supply your own.

The client validates the attestation bundle, logs the submission, streams response chunks, and finishes with summary=ok. That summary reports the request pipeline rather than the inference, so read the chunks too:

  • A payload the cloud app rejects also ends with summary=ok, carrying a single chunk whose text names the failure, such as inferenceRunner failed. error=missingPromptRequest.

  • A successful run streams many chunks, and a lone chunk is the signal to read its text.

Those chunks otherwise print as <undecodable>, because a release build does not log response bodies; add --verbose-proto-logging, which makes the client decode them and print the generated text alongside the model identity, the rendered prompt with its token identifiers, and the response’s own transparency report. For a liveness check that doesn’t depend on the model being up, fetch the node’s attestation instead. The connection can intermittently fail with closed the connection before completing the HTTP/2 connection preface; retry if this occurs.

On a research node, the client logs the softwareTransparency and firmwareTransparency claims as unverified, leaves certificateTransparency nil, and still reports Successfully validated attestation bundle. This is expected for a research node, not a tooling error; a transparency-proof failure explains why.

If the request fails with unavailable: "Service not ready", the model is not up. The node rejects the request before decrypting it, so it processes no request data; see Diagnose the workload.

Build the request payload yourself

Where prompt options don’t reach, --payload takes a complete generate-request JSON document — either a JSON string or a path to a .json file — carrying the chat prompt and the inference identifier that selects the model. A minimal payload asks the model a single question:

{
    "generateRequest": {
        "modelConfig": {"inferenceId": "com.apple.fm.language.instruct_server_v2.pcc_interact_pro"},
        "promptRequest": {
            "version1": {
                "prompt": {
                    "chat": {
                        "version1": {
                            "messages": [
                                {"version1": {
                                    "rolePrompt": {"system": {}},
                                    "prompt": {"components": []}
                                }},
                                {"version1": {
                                    "rolePrompt": {"user": {}},
                                    "prompt": {"components": [
                                        {"value": {"string": "What model are you?"}}
                                    ]}
                                }}
                            ]
                        }
                    }
                }
            }
        }
    }
}

The system message carries no components, so the model runs against its own default system prompt. Save the document as generateRequest.json and send a request straight to the compute node, bypassing the agent and proxy tiers:

% pcc-curl cbcurl \
    --hostname 127.0.0.1 --port 4442 \
    --payload generateRequest.json

--inference-id overrides the identifier in the payload’s modelConfig, so one document can serve several models.

pcc-curl also builds a prompt-template request, from --prompt-template-id with --prompt-template-variable or from a payload carrying promptTemplate in place of promptRequest. That form doesn’t apply here: a Google Cloud node’s cloud app requires a promptRequest and rejects a template with inferenceRunner failed. error=missingPromptRequest. The template form serves cloud apps that resolve templates server-side, such as those on Virtual Research Environment inference nodes.

Exercise the proxied request path

A request can reach a Google Cloud inference node through an Apple silicon proxy node using Transitive Trust. The research environment has no PCC Gateway, so pcc-curl simulates one in-process: it registers the Google Cloud instance as the only compute node and, when the proxy asks for a node, returns that instance’s attestation. Exercising the path end to end means driving both hops from your host: a PCC Agent instance in the Virtual Research Environment at its local address, and the Google Cloud instance through the tunnel.

Create the agent instance from a TIE Proxy PCC Agent release, as described in Create a workload. A running state with an assigned address means only that the VM booted, so wait for its cloud board to report healthy before sending a request:

% pccvre instance diagnostic -N <agent-instance> -- get-cloud-board-health

Then send the request to the agent instance, naming the Google Cloud node as the compute tier:

% pcc-curl cbcurl \
    --hostname <agent-address> --port 4442 \
    --compute-hostname 127.0.0.1 --compute-port 4442 \
    --workload-parameter inference-id=<inference-id> \
    --user-prompt "What model are you?"

Setting --compute-hostname makes --hostname and --port the proxy endpoint and defaults the workload type to pcc-agent, so you don’t have to name it. The inference identifier passed as a workload parameter is a routing key the agent’s own routing table consumes, distinct from --inference-id, which sets the payload’s modelConfig. A request whose identifier the routing table doesn’t know falls back to the passthrough agent, which still forwards the request to the compute node and returns a full response.

A proxied run logs Unknown message type received at error level for each proxy progress message (the response identifier, per-component timings, and Request execution log entries). These are not failures, and the request execution log is visible only through those error-level lines.

The resulting response carries both nodes’ attestations, so this is the flow that produces the mixed evidence a single request execution log can contain. The client runs a validator per hop, naming CloudAttestationValidator for the agent’s bundle and CloudAttestationV2Validator for the Google Cloud node’s.

Inspect an attestation bundle

A node’s attestation binds the measurements of the software it runs to its Google Cloud Roots of Trust; Google Cloud Node Attestation covers how the evidence is produced and bound together. A node produces an attestation each time it generates an attested key, so the requests it serves while that key is live share one bundle. A fetch returns the bundle for the node’s current key. pcc-curl keeps no bundle between runs, so every request it sends starts with a fetch.

pcc-curl validates the bundle it receives on the request path, but it evaluates no environment policy: it takes no environment option, and it proceeds when transparency claims come back unverified. Deciding whether to trust a bundle is done by pccvre, so inspecting one requires fetching it with pcc-curl and passing it to pccvre.

Fetch the bundle as raw protobuf, not JSON. pccvre inspects the protobuf version marker to route the bundle to the correct decoder. Because JSON cannot be inspected this way, the JSON path decodes the input in-process as a legacy bundle and rejects a current one.

% pcc-curl fetch-attestation \
    --hostname 127.0.0.1 --port 4442 --protobuf \
    --log-level critical > attestation.pb

--protobuf writes the binary bundle to standard output, but the client’s log lines go there as well, so a plain redirect prepends them to the bundle and pccvre rejects the result with read attestation bundle: malformedProtobuf. Silencing the log with --log-level critical leaves a clean file of roughly 97 KB. An empty file means the fetch failed; retry.

pccvre attestation parse converts the bundle to JSON without deciding whether to trust any of it. The dump names the bundle kind, gcpv1, and carries the three attestations a node produces — the Workload VM and Key Protection Service VM TDX quotes, and the host’s TPM quote with its Titan endorsement — alongside the attested metadata:

% pccvre attestation parse \
    --pccvre-attestation-helper "$HELPER" \
    -I attestation-bundle:protobuf attestation.pb

Each quote and boot event log stays Base64-encoded in that dump, so parsing alone does not decode the measurement registers, the report data, or the trust-domain attributes; Appendix: Google Cloud Measurement Registers covers those registers.

pccvre attestation verify additionally checks the bundle against an environment’s policy, and it’s the command that reports the attested key:

% pccvre attestation verify \
    --pccvre-attestation-helper "$HELPER" \
    -E dev -I attestation-bundle:protobuf attestation.pb

pccvre decodes legacy bundles in-process but hands a current one to pccvre-attestation-helper, which ships in a release’s PrivateCloudTools disk image rather than in macOS. Mount the image for the build under test and pass the path to the helper binary; the --tools-directory option is for the instance commands and doesn’t apply. Omitting --pccvre-attestation-helper makes pccvre fetch and cache the latest host-tools image from the transparency log instead.

% hdiutil attach -nobrowse -readonly ~/Downloads/<build>_PrivateCloudTools.dmg
% HELPER="/Volumes/<build>.PrivateCloudToolsContent/usr/bin/pccvre-attestation-helper"

Read a transparency-proof failure correctly

pccvre defaults to the production environment, whose policy rejects a node running software that was never published to the transparency log:

verification failed against production policy
{"error":"rejected by policy: firmwareTransparency claim is missing, unverified,
 or expired; softwareTransparency claim is missing, unverified, or expired"}

That is the correct verdict for unpublished software, not a tooling failure. Research instances typically run unpublished builds and report missingComputeNodeProofs, so pass -E dev, which matches the environment in the node’s own attestation. The development policy checks the bundle’s structure and leaves the transparency claims unbound. To attempt the stricter binding, drive the helper directly and have it fetch fresh proofs from the live log. Note that the helper takes the long --environment option rather than pccvre’s -E:

% "$HELPER" attestation verify -i binary --environment dev \
    --verify-transparency-proofs attestation.pb

That passes only once the node’s compute-node proofs are published. Before it reaches the proof-verification step, the helper checks certificate transparency, so on a node whose platform identity certificate is absent, it stops earlier, with Bundle has no platform certificate chain; cannot verify certificate transparency. The same gap shows up in pcc-curl‘s own log, at its default level, as No platform leaf certificate available; certificateTransparency claim left nil. A research node is not expected to carry a platform identity certificate, because it’s not part of the production fleet, so read this as a property of the environment rather than a fault in the node or the tooling.

A genuinely verifiable pass under the production policy therefore needs a published release whose proofs are in the log; auditing that log is the same on Apple silicon and Google Cloud, and Inspecting Releases describes it. For recomputing a node’s published measurements yourself, see Reproducing Google Cloud Measurements.

Inspect diagnostic logging

Three vantage points cover a node’s diagnostic output, in increasing order of what they require.

The serial port needs no shell access and is the primary source for boot and workload-startup problems. The launcher writes its log to the serial port. On a debug image, setting tee-container-log-redirect=serial in the instance’s metadata sends the workload’s standard output and error to the serial port as well:

% gcloud compute instances get-serial-port-output "${INSTANCE_NAME}"

Search the output for launcher lines to confirm which workload image was pulled and started, including its digest. Startup output appears early in the log, so an instance that fails to start usually logs the cause there.

The container journal needs a shell in the Workload VM and reaches the workload’s own systemd units. The workload container is named tee-container:

% sudo ctr tasks exec --exec-id debug tee-container journalctl -fu <service-name> 2>&1

A shell inside the container gives you everything else:

% sudo ctr tasks exec --exec-id debug --tty tee-container /bin/bash

ctr commands need root in the Workload VM. The --exec-id value is arbitrary but has to be unique per concurrent exec. --tty needs an interactive SSH session; it doesn’t attach over a one-shot non-interactive ssh invocation.

Diagnose the workload

Confirm the container is running before anything else:

% sudo ctr tasks ls
TASK                       PID     STATUS
guest-gpu-tools-task       7134    RUNNING
workload-service-daemon    7488    RUNNING
tee-container              7718    RUNNING

An empty task list with a container present means the launcher created the container but never started it, or the container has since exited; check the launcher’s own journal with sudo journalctl -u container-runner. After a reboot or a re-image, expect the whole workload to be absent for a while — no tasks, no containers, and no launcher journal entries yet — while the instance brings it up. If SSH times out during the banner exchange in that window, check uptime: the instance is still coming up.

Inside the container, systemctl status shows the overall state, and systemctl list-units --failed lists the failed units. Two workload-specific readings matter. Queued jobs mean some units have not attempted to start yet, so if you read the state too early, degraded can point at a unit that systemd never tried to start. A failed unit stays failed across a restart attempt until systemctl reset-failed clears it; that persistence is the usual reason a component doesn’t come back after a patch.

Reach a component container

Components inside the workload container run as their own runc containers, launched by systemd units. To access one, use runc exec instead of ctr. Container identifiers follow a <unit>-<hash> pattern, so match the prefix rather than copying a hash:

% runc exec "$(runc list -q | grep '^cloudboard-' | head -1)" <command>

If runc list is empty, runc is using a non-default state directory. Read the --root value from the unit’s ExecStart line with systemctl cat <unit> and pass it through.

Inspect telemetry egress

Audit configuration and audit lists inside the measured container govern what telemetry may leave the node. You can see exactly what crosses that boundary by redirecting the node’s telemetry to a receiver you run and control, then reading what arrives.

The telemetry pipeline takes its endpoints from unmeasured configuration, so you can redirect it without rebuilding the image or editing files on the node. A debug flag turns off peer-certificate verification for the outputs, which is what lets an ad-hoc receiver terminate the connections.

Redirecting telemetry changes where the data goes, not what may leave. The measured audit configuration governs what may leave, and it stays independent and attestable, so you can run with auditing enforced and the environment set to customer and still observe what crosses the boundary. The flag affects only whether an external receiver’s certificate is verified: traffic stays encrypted, but flows to an unauthenticated endpoint outside the boundary. The shipped default has verification enabled, so a node that doesn’t set the flag is unaffected.

With that boundary in mind, set the endpoints and the flag as unmeasured configuration for the telemetry component:

OTEL_SPLUNK_ENDPOINT: localhost
OTEL_MOSAIC_ENDPOINT: localhost
OTEL_PRISM_ENDPOINT: localhost
PCC_TELEMETRY_INSECURE: "1"

Run a receiver that listens on the ports the node sends to and prints what it receives. The workload container shares the Workload VM’s network namespace, so localhost from inside the container reaches the receiver on the VM. Then watch both ends:

% docker logs -f fluent-receiver
% sudo ctr tasks exec --exec-id logs tee-container \
    journalctl -u fluent-bit.service -f | grep -E 'status=|verify|INSECURE'

Expect HTTP status=200 from each output and no certificate-verification failures. Before the receiver is listening, the sender logs error initializing http request on every output; that clears once the receiver starts. The receiver renders metrics and traces in a structured non-JSON form, so searching its output for JSON strings misses them. The sender-side status is the reliable signal for those, while log records show up at the receiver directly.

Build the receiver

The receiver terminates TLS on all three of those ports and prints what arrives. This Dockerfile is the entire build context; the configuration and the entry point are inlined, so there are no accompanying files:

# The -debug variant ships a shell and openssl; the default image is distroless
# and the entry point needs both to generate a certificate.
FROM fluent/fluent-bit:latest-debug

COPY <<'EOF' /fluent-bit/etc/fluent-bit.yaml
service:
  log_level: info
  http_server: true
  http_listen: 0.0.0.0
  http_port: 2020

pipeline:
  inputs:
    - name: splunk
      listen: 0.0.0.0
      port: 18088
      tag: splunk
      tls: on
      tls.verify: off
      tls.crt_file: /certs/server.crt
      tls.key_file: /certs/server.key

    - name: opentelemetry
      listen: 0.0.0.0
      port: 24184
      tag: otel_mosaic
      tls: on
      tls.verify: off
      tls.crt_file: /certs/server.crt
      tls.key_file: /certs/server.key
      buffer_max_size: 64M
      buffer_chunk_size: 1M

    - name: opentelemetry
      listen: 0.0.0.0
      port: 443
      tag: otel_prism
      tls: on
      tls.verify: off
      tls.crt_file: /certs/server.crt
      tls.key_file: /certs/server.key

  outputs:
    - name: stdout
      match: "*"
      format: json_lines
EOF

COPY --chmod=755 <<'EOF' /entrypoint.sh
#!/bin/sh
set -e
if [ ! -f /certs/server.crt ]; then
    mkdir -p /certs
    openssl req -x509 -newkey rsa:2048 -nodes \
        -keyout /certs/server.key -out /certs/server.crt -days 365 \
        -subj /CN=localhost -addext subjectAltName=DNS:localhost,IP:127.0.0.1
    chmod 644 /certs/server.key
fi
exec /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.yaml
EOF

EXPOSE 18088
EXPOSE 24184
EXPOSE 443

ENTRYPOINT ["/entrypoint.sh"]

Four details in the Dockerfile matter:

The ports match those configured on the node. Port 18088 carries logs, 24184 carries metrics, and 443 carries traces. The receiver stands in for the destinations those outputs already target, so it answers on the same ports.

Setting tls.verify: off on inputs disables client certificate requests. Combined with the sender-side flag, this allows an ad-hoc receiver to complete the handshake while keeping traffic encrypted.

Setting buffer_max_size: 64M on the metrics input accommodates the workload’s metrics flushes, which exceed the 4 MB default. Without it, the receiver returns HTTP 413, which gRPC reports as an opaque connection error without indicating a size issue.

The certificate is generated at launch, not at build time. This prevents a published image from carrying a stale certificate. Because the sender does not verify the certificate, it only needs to be valid TLS material; the entry point still sets a localhost subject and SAN so any tool checking it finds valid metadata.

Build it for the node’s architecture and push it to the registry your reservation provides, which is empty until your first push:

% container build --platform linux/amd64 \
    -t "${BOOKING_REGISTRY_URL}/fluent-bit-receiver:debug" --push .

In the Workload VM, authenticate to the registry with the metadata server’s token, then pull and run:

% TOKEN=$(curl -s -H 'Metadata-Flavor: Google' \
    http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token \
    | grep -o '"access_token":"[^"]*"' | cut -d'"' -f4)
% echo "$TOKEN" | docker login -u oauth2accesstoken --password-stdin <registry-host>
% docker pull "${BOOKING_REGISTRY_URL}/fluent-bit-receiver:debug"
% docker run -d --name fluent-receiver \
    -p 18088:18088 -p 24184:24184 -p 443:443 \
    "${BOOKING_REGISTRY_URL}/fluent-bit-receiver:debug"

The explicit docker login is necessary because the debug image ships neither gcloud nor a Docker credential helper, so nothing fetches a registry token on Docker’s behalf. The token lasts about an hour; re-run the login if a later pull returns 403. A reboot wipes both the container and the image cache, so expect to pull and run again after one; the certificate regenerates itself.