Google Cloud Workload Modification

Patch a component inside the measured workload container and deploy the result to a research instance.

Overview

A node’s runtime arrives as a single measured container image, the Workload Megacontainer. Changes made inside a running instance are unmeasured and don’t survive a restart. Changing what a node runs requires rebuilding the container image and re-imaging the instance. The node attests the rebuilt image, so the change appears in the node’s attestation.

Understand what you need for the modification

container

Apple’s container runtime for macOS. On first use, container system start may prompt you to download a default kernel; accept it, or container build and container run fail with a missing-kernel error.

Megacontainer build tool

The image embeds other images in a nonstandard way, so docker build alone cannot correctly unpack, modify, and repack it. That tool is pccvre-container-export, which operates directly on apple/container’s local content store and supports only the apple-container: transport. pccvre installs it for you, downloading it from the transparency-log entry for the most recent release.

gcloud

Authenticate with it against the registry you pull from and the one you push to.

skopeo and jq

Use them to read the raw image configuration, which container image inspect normalizes away.

Push access to a registry

Your reservation provides one.

Understand how the container is structured

A Megacontainer is a single OCI image whose file system embeds several other OCI images. Each embedded image starts as its own container inside the outer one, managed by systemd and runc. The build tool treats the outer image as a black box:

  • list: Enumerates the embedded components.

  • export: Extracts an embedded component as a standalone OCI image.

  • inject: Places a modified component back as an outer layer, which boot-time scripts load on the next start.

  • fix-config: Restores image metadata dropped during builds.

You cannot add a new component this way, and you cannot change a component’s declared entry point, restart policy, or other life cycle properties; those are fixed when the image is assembled, and changing them means modifying the outer image.

Edit a configuration file on a running node

Rebuilding is the only way to change the image, but on a node booted from a debug image you can edit a component’s configuration file in place for a quick experiment, once you Open an SSH session. Such edits are ephemeral: a component’s root file system is an overlay whose lower layer is read-only and whose writable upper layer is regenerated on every start, so an edit doesn’t survive a restart of the unit and you need to re-apply it each time. It also leaves the node out of step with its own attestation, which was taken at boot from the unmodified image, so reproduce anything you intend to report from a rebuilt, re-imaged node. Prefer unmeasured configuration when the desired setting supports it. The Measured, unmeasured, and dynamic configuration details which settings are measured, and Inspect telemetry egress demonstrates this approach. Edit the file only for fields you can’t set that way.

From a shell in the workload container, stop the unit and its path trigger, generate a fresh overlay, edit the writable copy, and run the component in the foreground:

% systemctl stop <unit>.path
% systemctl stop <unit>.service
% export INVOCATION_ID=$(cat /proc/sys/kernel/random/uuid)
% megacontainer-create <unit>
% sed -i 's/<old>/<new>/' \
    /var/lib/containers/run/<unit>-${INVOCATION_ID}/rootfs/<path-to-config>
% megacontainer-run <unit>

systemd normally supplies INVOCATION_ID, but here you’re driving the create and run steps by hand, so supply one yourself to make them share an overlay. The Megacontainer ships no editor; sed, cat, and tee are in the image and on its execution allow list. Stop the path unit as well as the service, or the path unit starts the service again while you work. If the unit already failed, run systemctl reset-failed first, and note that the unit’s stop-post step deletes the overlay, so recreate it rather than looking for the previous one. To iterate, repeat from the create step and re-apply the edit.

Extract a component

Authenticate to the source registry and pull the base image:

% gcloud auth print-access-token | \
    container registry login -u oauth2accesstoken --password-stdin \
    "$(echo "$REGISTRY_URL" | cut -d/ -f1)"
% container image pull --platform linux/amd64 "$MCIMAGE"

The --platform flag matters. These images are multi-architecture indexes, and apple/container defaults to arm64 on Apple silicon while the target Workload VM runs linux/amd64. Omitting this flag means you patch an arm64 component, which inject rejects on its architecture check.

List what is inside and confirm the component you intend to patch is there:

% pccvre-container-export list -i "apple-container:$MCIMAGE"
NAME
cb-attestationd
cloudassetdownloader
cloudboard
cpagent
lsecuritymonitorlited
...

Then pull it out as a standalone image:

% pccvre-container-export export -i "apple-container:$MCIMAGE" "$SERVICE" \
    -o "apple-container:service:extracted"

You don’t need to know the outer image’s layout to patch a component; the component’s file system is an ordinary container file system. Explore it before deciding what to overlay, passing --platform and overriding the entry point so the component’s own entry point doesn’t run instead of your shell:

% container run --rm -it --platform linux/amd64 --entrypoint /bin/sh service:extracted

Many of these images ship without a shell. In that case, overlay one in a new image, or export the file system and read it outside the container.

Patch and inject the component

Write a Dockerfile that starts FROM the extracted component and applies your change. Anything you can do to a regular image works: COPY overlays files and RUN patches them in place. For example:

FROM service:extracted
RUN echo "PATCHED" > /PATCHED

Build it, sanity-check it standalone, and inject it back:

% container build --platform linux/amd64 -t service:patched --file service.Dockerfile .
% container run --rm service:patched /bin/sh -c "cat /PATCHED"
% pccvre-container-export inject -i "apple-container:$MCIMAGE" "$SERVICE" \
    "apple-container:service:patched" \
    -o "apple-container:mc:patched"

inject verifies the component exists in the base image, checks that your patch matches its architecture and operating system, and appends a new outer layer that overlays the component’s blobs. It doesn’t rewrite the image; appending rather than rewriting is safer for iterative work but grows the image slightly. The result is a fully formed image in which the target component boots from your modified content.

Restore dropped image metadata

container build doesn’t propagate .config.ExposedPorts or .config.Volumes from the parent image through FROM, whereas Docker, BuildKit, podman, and buildah all do. The symptom is subtle: the image boots correctly and its units come up, but requests to workload ports hang or are refused because the launcher reads exposed ports to configure routing and finds the field empty.

Restore the fields from the pre-patch image:

% pccvre-container-export fix-config \
    --from "apple-container:$MCIMAGE" \
    "apple-container:mc:patched" \
    -o "apple-container:${DESTIMAGE}"

Verify they survived. container image inspect shows a normalized view that hides whether these fields are stored on disk, so read the raw OCI configuration:

% container image save "${DESTIMAGE}" | \
    skopeo inspect --config --override-arch amd64 --override-os linux \
    oci-archive:/dev/stdin | jq '.config | {ExposedPorts, Volumes}'

Both fields need to match what the base image declared. An explicit EXPOSE or VOLUME line in a Dockerfile is honored; only the inheritance path is broken, so fix-config works by generating those lines for you. Write the Dockerfile yourself when you want to combine the metadata restore with the outer-image overlays below in a single build.

Modify measured configuration

Some configuration settings reside on the outer image’s file system rather than inside an individual component. Files under /mc/config/ define per-component configuration that participates in the attested image identity: the runtime includes anything under that path in the measurement reported at attestation time. The layout nests by component and PCC variant, the name of the configuration profile the node is running. Each file’s shape is component-specific:

/mc/config/<component>/<pcc-variant>/measured-config.json

A PCC variant selects one of several configuration profiles, and the runtime measures the selected profile into the attestation. A VRE variant takes the model and prompt archive sources from the control plane instead of from measured configuration, and otherwise keeps the production security configuration, including the production host image. VRE variants don’t run in production. Because the runtime measures the selected variant, a node running a VRE variant attests differently from a production node, so its attestation doesn’t verify against a production policy. Read a transparency-proof failure correctly describes the same effect from the verifier’s side. The workload container carries configuration for every variant.

Overlay a modified file by placing it at the corresponding path in your working directory and adding a COPY to the outer Dockerfile, alongside the EXPOSE and VOLUME lines from the previous section:

FROM mc:patched

EXPOSE 4442
VOLUME ["/var/lib/containers/overlays", "/var/lib/containers/run"]

COPY overlays/measured-config.json /mc/config/<component>/<pcc-variant>/measured-config.json

Building that Dockerfile replaces the fix-config step, so tag its output $DESTIMAGE to match what the deploy section expects:

% container build --platform linux/amd64 -t "$DESTIMAGE" --file outer.Dockerfile .

This is a boundary worth exercising deliberately. Configuration inside the image is measured and covered by attestation; configuration the control plane delivers is not part of the release, and PCC treats it as attacker-controlled, so it must not change the node’s security or privacy properties. For more details on the split, see Measured, unmeasured, and dynamic configuration. Change a measured setting here, deploy, and confirm that the node’s attestation changes.

Update the execution allow list

lsecuritymonitorlited enforces an Execution enforcement that maps each permitted path to the SHA-256 digest of its contents, so at runtime it refuses any binary whose digest is not listed, even when that binary exists on disk and is marked executable. The primary list is /mc/eh-allowlists/base.json, with a per-component file under /mc/eh-allowlists/components/:

{
  "version": 1,
  "entries": {
    "/usr/bin/bash": "sha256:97995faa249e5706dd0b0373c9da547709bf7349755d5fc8e52f97a4bd04feaf"
  },
  "scriptEntries": {
    "/opt/afm_runtime/triton_repos/v11_pro/causal_lm/1/model.py": "sha256:<digest>"
  }
}

Keys are absolute paths as they appear inside the component’s own runtime container; values are digests of the file’s contents. Binaries go in entries; Python sources and bytecode go in scriptEntries, which the Script-execution policy enforces. Update the map that matches the file when you:

  • Patch a file, causing its digest to change.

  • Add a file with no existing entry.

  • Permit a file the policy disallows.

Start from the list in the image itself rather than a reference copy, so the list matches what is live. Then add your entry and overlay the file:

% container run --rm mc:patched cat /mc/eh-allowlists/base.json > overlays/base.json
% shasum -a 256 path/to/binary
COPY overlays/base.json /mc/eh-allowlists/base.json

Patching a component binary without updating the allow list is a common failure. The image boots, systemd starts the unit, and the process fails immediately when it attempts to execute the modified binary, usually reporting a permission or exec refusal in that unit’s journal. After deploying, confirm the two digests agree:

% grep <binary> /mc/eh-allowlists/base.json
% runc exec "$(runc list -q | grep "^${SERVICE}-" | head -1)" sha256sum /usr/bin/<binary>

Push and deploy

Authenticate to the destination registry if it differs from the source, then push the image you tagged $DESTIMAGE:

% gcloud auth print-access-token | \
    container registry login -u oauth2accesstoken --password-stdin \
    "$(echo "$BOOKING_REGISTRY_URL" | cut -d/ -f1)"
% container image push "$DESTIMAGE"

Then re-image the instance with it, as described in Replace the workload container. On the next boot the launcher pulls your image; the serial-port output shows the digest it pulled, and runc exec inside the running container confirms your change landed. Verify the resulting measurement in the node’s attestation before trusting the swap: Inspect an attestation bundle.

pccvre-container-export reads and writes only apple/container’s local store, so to take a patched image out as a tar archive, run container image save after the final step.