Google Cloud Research Setup

Configure your host, reach a research instance, and change what that instance runs.

Overview

A Google Cloud research instance runs on real cloud hardware, and three of its properties shape everything below:

  • You obtain an instance in one of two ways: Apple provisions one and reserves it for you, or you create one yourself in a project you own.

  • A reserved instance isn’t directly reachable: every connection goes through a tunnel. On an instance you create, the network path is yours to configure.

  • You don’t configure the instance in place: the configuration that matters is inside the measured workload container, so changing it means replacing the image the instance boots.

Install the prerequisites

Install the following on your host:

gcloud

The Google Cloud command-line tool. It authenticates you, establishes tunnels, and performs instance life-cycle operations.

pccvre

Ships with macOS under /System/Library/SecurityResearch/usr/bin. Use it to audit the transparency log, as described in Inspecting Releases, and to parse and verify a node’s attestation.

container

Apple’s container runtime for macOS, needed to unpack, modify, and run the workload image. Complete its Understand what you need for the modification before you start.

An SSH key pair

Register the public key when you reserve an instance, and configure your local ssh client to use the private key. Shell access requires a debug image; see Debug-image policies.

Choose how to get an instance

The two paths differ in what you control and in how much of the stack the instance can attest to:

A reserved instance

Apple provisions an instance with GPUs attached, on the machine type that serves production traffic, and reserves it for a period. Its attestation covers the whole stack, GPU measurements included. Capacity is limited, and the reservation pins the instance to one machine.

An instance you create

You run the workload on a machine type with no GPU attached, in your own project, with credentials you hold. Capacity is readily available. The workload container needs rebuilding first, because the shipped image does not boot without a GPU; Running the Workload Without a GPU on Google Cloud covers that rebuild and the gcloud command that launches the instance. The attestation is narrower: no GPU measurements, and no host measurements on a machine type whose firmware exposes no TPM.

The rest of this article applies to both, except where it says otherwise. A reservation supplies the project, zone, service account, and instance name the commands below use; on an instance you create, you set those values yourself.

Configure access to a reservation

Capacity for instances with GPUs attached is limited, so they come from time-based reservations rather than on demand. A reservation gives you an instance name, a project, a zone, a service account to impersonate, an SSH user name, and container registries you can pull from and push to. You receive these as a shell fragment that you save as booking.env and source, to simplify gcloud command invocations:

% source booking.env

Create a named gcloud configuration for the reservation so that later commands need no per-invocation flags:

% gcloud config configurations create "vre-${BOOKING_ID}"
% gcloud config set core/project "${PROJECT}"
% gcloud config set compute/zone "${ZONE}"
% gcloud config set auth/impersonate_service_account "${SERVICE_ACCOUNT_EMAIL}"
% gcloud config configurations activate "vre-${BOOKING_ID}"

Every command in this section and the two that follow assumes the reservation’s configuration is active; without it, add --project, --zone, and --impersonate-service-account to each gcloud invocation. On an instance you create yourself, set up a similar configuration with your own project and zone, omitting the impersonation setting if you authenticate directly.

Reach an instance

Reserved instances are not directly reachable from the public internet. Google’s Identity-Aware Proxy mediates both the request data path and the SSH connection, and confines access to an authenticated Google identity. You open a tunnel before connecting with any other tool. On an instance you create, this is one option among the network configurations you can choose, and the rest of this article assumes it.

Bind the instance’s workload port to a local port. The workload listens on 4442, which is the default the request tooling expects, so binding the local end to the same number keeps later commands short:

% gcloud compute start-iap-tunnel "${INSTANCE_NAME}" 4442 \
    --local-host-port="127.0.0.1:4442" &
% TUNNEL_PID=$!

The tunnel runs in the background; kill $TUNNEL_PID closes it. Tunnels to the same instance can coexist on different local ports, so a data-path tunnel and an SSH tunnel run side by side.

The impersonated service account authenticates the tunnel and needs roles/iap.tunnelResourceAccessor on the instance. Without that role, the tunnel command fails with a permission error rather than hanging, which distinguishes it from an instance that is not yet reachable.

Open an SSH session

Shell access requires a debug image; a production-configured instance rejects SSH by design. On a debug image it is ordinary SSH, with gcloud supplying only the tunnel and ssh authenticating with your local key:

% ssh -o "ProxyCommand=gcloud compute start-iap-tunnel ${INSTANCE_NAME} %p \
        --listen-on-stdin --zone=${ZONE} --project=${PROJECT} \
        --impersonate-service-account=${SERVICE_ACCOUNT_EMAIL}" \
    -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
    "${SSH_USERNAME}@localhost"

This lands you inside the Workload VM, the confidential VM that runs the workload, rather than on the host underneath it. The keys you registered are provisioned in the Workload VM, which is why SSH responds only once that VM is up, well after the instance reports RUNNING. The reservation’s user belongs to the docker and google-sudoers groups, so docker and sudo work there. For what to run once you are in, see Diagnose the workload.

Confirm from the kernel’s view which environment you reached. The output also confirms that Intel TDX is active:

% sudo dmesg | grep -i tdx
tdx: Guest detected
tdx: Attributes: SEPT_VE_DISABLE
systemd[1]: Detected confidential virtualization tdx.

Reach the host

Some things you need to inspect are outside the Workload VM: the hypervisor’s own view of the confidential VMs it runs, and any disk attached to the instance rather than passed through to the VM. Reaching the host requires a second SSH hop that you make yourself, from inside the Workload VM.

Enable the hop before the boot where you want to use it, by setting tee-debug-ssh-enable in instance metadata. The host’s oem-entrypoint.service reads that at boot, mints an ephemeral SSH key pair, and shares the private key with the Workload VM through the firmware-configuration channel. The service mints a new key on every boot, so the key doesn’t persist across a reboot:

% gcloud compute instances add-metadata "${INSTANCE_NAME}" \
    --metadata="tee-debug-ssh-enable=true"
% gcloud compute instances stop "${INSTANCE_NAME}"
% gcloud compute instances start "${INSTANCE_NAME}"

Stop and start the instance, allowing for the reservation-release window that Change what an instance runs describes. Rebooting from inside the Workload VM restarts only the VM, so the host never rereads the metadata.

Once the instance is back up, hop from the Workload VM to the host using the private key the host left for you:

% sudo ssh -i /sys/firmware/qemu_fw_cfg/by_name/opt/tee_debug_ssh/id_rsa/raw \
    root@192.168.100.1

To confirm you are on the host, look for the hypervisor process running the confidential VMs; the Workload VM cannot see it:

% ps aux | grep -i /usr/local/bin/qemu-system-x86_64

A running Workload VM appears as a qemu-system-x86_64 process with guest=WorkloadVM in its arguments.

This is a debug-image capability. A production-configured instance grants no such path.

Run commands over a persistent tunnel

For repeated non-interactive commands, split the work: hold one tunnel open on a fixed local port, then run plain ssh against it. Each command then reuses a single tunnel instead of spawning gcloud per invocation, and the ssh invocation becomes a stable prefix you can add to an allow list once:

% gcloud compute start-iap-tunnel "${INSTANCE_NAME}" 22 \
    --local-host-port="127.0.0.1:2222" &
% ssh -p 2222 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
    "${SSH_USERNAME}@127.0.0.1" -- 'uname -a'

Drop the trailing -- '<command>' for an interactive shell, and add -t when the command needs a terminal.

Change what an instance runs

To change what runs on a PCC instance, change the image it boots and restart it. Two things are separately replaceable: the workload container and the host image on the boot disk.

Both procedures stop and start the instance, and on a reservation-bound machine those are not symmetric operations. The reservation pins the instance to a specific host and holds that machine for a period — in practice around 15 minutes — after the instance releases it, so a start issued straight after a stop fails with a capacity error. Wait out the window before starting. An instance you create yourself is not pinned this way, so a start straight after a stop succeeds. Once started, an instance can take upwards of 40 minutes to become reachable: the instance reports RUNNING and the serial console shows the Workload VM booting long before SSH answers, so neither is a readiness signal.

Replace the workload container

The workload launcher, the container-runner service that pulls and starts the measured container, reads the workload-container reference from instance metadata at boot, so updating the reference takes effect on the next start:

% gcloud compute instances stop "${INSTANCE_NAME}"
% gcloud compute instances add-metadata "${INSTANCE_NAME}" \
    --metadata="tee-image-reference=${IMAGE_REFERENCE}"
% gcloud compute instances start "${INSTANCE_NAME}"

To read the currently configured reference without touching the instance:

% gcloud compute instances describe "${INSTANCE_NAME}" \
    --format="value(metadata.items['tee-image-reference'])"

Confirm from the serial-port output which image actually launched, as described in Inspect diagnostic logging. Google Cloud Workload Modification covers building a reference to deploy here, and Running the Workload Without a GPU on Google Cloud covers building one that boots on a machine with no GPU attached.

Replace the boot disk

Swapping the boot disk moves the instance to a different host image, most often between the hardened image that a production-configured instance boots and the debug variant that permits shell access. The instance itself survives, so its IAM bindings, network interfaces, and service account are preserved.

Match the new disk’s size and type to those of the disk you are replacing. The size has to be at least the image’s own size:

% gcloud compute instances stop "${INSTANCE_NAME}"
% CURRENT_DISK=$(gcloud compute instances describe "${INSTANCE_NAME}" \
    --format="value(disks[0].source)" | awk -F/ '{print $NF}')
% gcloud compute disks create "<new-disk>" \
    --image-project="<image-project>" \
    --image="<image-name>" \
    --type=hyperdisk-balanced --size=100
% gcloud compute instances detach-disk "${INSTANCE_NAME}" --disk="${CURRENT_DISK}"
% gcloud compute instances attach-disk "${INSTANCE_NAME}" \
    --disk="<new-disk>" --boot
% gcloud compute instances start "${INSTANCE_NAME}"

A disk larger than its source image draws a warning that the root partition may need manual resizing. Matching the disk you replace keeps you on a size at which you know the image can boot.

The old disk survives the detach, so reverting means repeating these steps with the old disk name. Delete it with gcloud compute disks delete only once you are confident in the new one.

Inspect a hardened host image

A hardened host image is production-configured and ships no remote terminal. A research instance that Apple provisions carries a second disk built from that image alongside the debug image on its boot disk, allowing you to inspect the hardened file system from the debug environment. On an instance you provision yourself, attach that disk explicitly.

That second disk, the inspection disk, is attached to the instance, so it appears on the host rather than inside the Workload VM. Reach the host first, then find the disk there. It appears as a second NVMe namespace alongside the boot disk, and the namespace number gives it a stable path:

% nvme list
Node          SN            Model         Namespace  Usage
/dev/nvme0n1  nvme_card-pd  nvme_card-pd  1          107.37 GB / 107.37 GB
/dev/nvme0n2  nvme_card-pd  nvme_card-pd  2           31.14 GB /  31.14 GB
% ls -l /dev/disk/by-id/nvme-nvme_card-pd_nvme_card-pd_2

Namespace 1 is the boot disk and namespace 2 is the inspection disk; their sizes tell them apart. On a disk attached before boot, the device name set at attach also appears, as /dev/disk/by-id/google-<device-name>. A disk hot-attached to a running instance may not get that symlink, so the namespace path is the safer default.

Mount the partition labeled ROOT-A, read-only. A read-only attach is not available on this machine type: hyperdisk-balanced rejects one, and the machine type rejects the pd-balanced type that allows one. Nothing at the hypervisor layer therefore protects the disk from writes, so only the -o ro mount keeps your inspection nondestructive, and noload suppresses the journal replay that otherwise writes to the disk before you read it. The host’s own root file system is read-only, so put the mount point somewhere writable such as /tmp:

% lsblk -f /dev/nvme0n2
% mkdir -p /tmp/hardened
% mount -o ro,noload /dev/nvme0n2p3 /tmp/hardened

Confirm the partition number against lsblk -f rather than assuming it: the layout also carries a STATE partition, an EFI-SYSTEM partition, and a second unlabeled ext4 file system.

Start with what the image does not contain. A hardened host image ships a shell (/bin/bash, with /bin/sh linked to it), but runs no remote service that exposes it, and its console login has no usable account: /etc/shadow holds only a disabled root entry. Search for remote-access daemons, then check which units the image enables at boot; the presence of a shell tells you nothing about either. The host has no find, so list the directories and filter with grep:

% ls /tmp/hardened/usr/sbin /tmp/hardened/usr/bin /tmp/hardened/sbin \
    /tmp/hardened/bin /tmp/hardened/usr/libexec /tmp/hardened/usr/lib/ssh \
    /tmp/hardened/usr/lib/openssh 2>/dev/null | grep -iE 'sshd|telnet|dropbear'
% ls /tmp/hardened/etc/ssh
% ls /tmp/hardened/usr/lib/systemd/system/ | grep -iE 'ssh|debug-shell'
% ls /tmp/hardened/usr/lib/systemd/system/multi-user.target.wants/
% ls /tmp/hardened/usr/lib/systemd/system/sockets.target.wants/
% ls /tmp/hardened/etc/systemd/system/

The daemon is absent, not idle: there is no sshd binary in any of those directories, no host keys under /etc/ssh, and no unit or socket that starts a listener. Judge by the binary and the keys rather than by the configuration, because the image still ships an sshd_config. Not every directory in that first command exists on the image, which is why it discards errors. Listing /usr/lib/systemd/system shows which SSH and debug-shell units the image ships at all, and the *.target.wants directories and /etc/systemd/system show what it starts at boot. The distinction shows up in debug-shell.service, which ships without being enabled.