--- title: Running the Workload Without a GPU on Google Cloud framework: Private Cloud Compute Security Guide role: article role_heading: Article path: private-cloud-compute/googlecloudworkloadwithoutgpu --- # Running the Workload Without a GPU on Google Cloud Overlay the measured workload image with a stand-in model, so the node boots and serves inference on CPU alone. ## Overview Overview Most of a node’s software is independent of inference running on GPUs. Attestation, the encrypted request path, configuration delivery, the execution allow list, and telemetry egress run on the CPU, and you can exercise all of them on an instance with no GPU attached. Instances with GPUs attached are scarce and come from a reservation, so this is the path open to a researcher who cannot reserve one: you build the image, create the instance yourself on a widely available machine type, and hold every credential involved. It works the same way if you do have a reservation. Two hard dependencies stop the shipped image from booting without a GPU: three components need a GPU driver to reach a ready state, and the model repositories the image ships load a back end that imports CUDA. Rebuilding the workload image around a no-op model, a stand-in that streams synthetic tokens and computes nothing, removes both. The result reaches the same end state as a GPU node, less the GPU components: cpagent, cloudboard, cb-attestationd, cloudassetdownloader, ensemble-transport, and afm-runtime running, and a request from Issue inference requests returning a complete generation over an attested, encrypted connection. Everything here is an overlay on the outer image. Unlike Google Cloud Workload Modification, you don’t extract, patch, or inject any embedded component, so you don’t need the Megacontainer build tool; an ordinary container build is enough. The code listings below take the base workload container image reference from the $MCIMAGE variable, as Understand what you need for the modification describe. Three components share the work of executing an inference request: Modify the workload image to run without a GPU To adapt a published AFM workload container image for CPU execution, update these six components: Every change above sits inside the image, so each one alters the workload container digest, and with it the measurement the node attests to. Reuse the repository’s own tensor contract The inference server rejects any request that names a tensor its model configuration doesn’t declare, and that declaration set moves between builds. When you write your own configuration, you need to keep it in sync by hand, and a mismatch appears one tensor at a time as unexpected inference input/output at request time. Avoid that sync burden by replacing only model.py and leaving the repository’s configuration alone. Use v11_pro of the six repositories; its causal_lm configuration declares the tensors tie-cloud-app sends, including the prompt-cache-warmup input that the other repositories omit, and its instance group is already KIND_CPU. Point afm-runtime at that repository and bind-mount your model over the shipped one, preserving the tensor contract from that build. Understand the no-op model tie-cloud-app requests two output tensors, output_ids and request_metrics, so those are the only two the stand-in produces. It streams one response per token until it emits the requested count, and attaches a metrics tensor to the last response. Two directory names differ by one character throughout this article: the hyphenated causal-lm/ is the staging directory on your host, and the underscored causal_lm/ is the repository directory inside the image that you mount over. Write this as causal-lm/model.py: import json import numpy as np import triton_python_backend_utils as pb_utils # 50 and 106 are the stop-token ids the caller hardcodes. Emitting one ends # the stream early and misreports why generation stopped. FILLER = tuple(i for i in range(1000, 1016) if i not in (50, 106)) class TritonPythonModel: def initialize(self, args): self.model_config = json.loads(args["model_config"]) def execute(self, requests): for request in requests: sender = request.get_response_sender() length = pb_utils.get_input_tensor_by_name(request, "request_output_len") count = max(1, int(np.asarray(length.as_numpy()).reshape(-1)[0])) for index in range(count): last = index == count - 1 # (batch, hypotheses, tokens) tensors = [pb_utils.Tensor( "output_ids", np.array([[[FILLER[index % len(FILLER)]]]], dtype=np.int32))] if last: metrics = json.dumps({"cached_token_count": 0}).encode() tensors.append(pb_utils.Tensor( "request_metrics", np.array([[metrics]], dtype=np.object_))) sender.send( pb_utils.InferenceResponse(output_tensors=tensors), flags=(pb_utils.TRITONSERVER_RESPONSE_COMPLETE_FINAL if last else 0)) return None This code imports only the standard library, numpy, and the Python back end’s helper module, all of which the image already provides. The model uses Triton’s decoupled transaction policy, which allows a single request to yield multiple responses through the response sender; therefore, execute returns None rather than a list. The token IDs are arbitrary, so the client receives nonsense text. The metrics structure fields are optional, so any valid JSON object decodes. Generation stops once the model emits the number of tokens tie-cloud-app asked for, and tie-cloud-app reports that it reached the output-length limit. note: The stand-in streams synchronously, so a second concurrent request waits behind the first. Choose a tokenizer tie-cloud-app resolves a tokenizer based on the identifier declared in a resource bundle, then loads the file from a hard-coded path, /weights/tokenizer.model. It doesn’t check the file against either the identifier or a digest. A SentencePiece model parses whichever file you stage. Because the no-op model discards the IDs it receives, the choice of tokenizer doesn’t affect whether generation completes. Use the tokenizer that ships with Gemma 4, one of Google’s open models; it’s a single tokenizer.model file in the model download. The setup doesn’t run a real model, so the specific tokens don’t matter; don’t read the resulting token stream as evidence of input tokenization. You cannot bake the tokenizer file into the image at /weights. The container launcher mounts a tmpfs there, which shadows whatever the image holds at that path, so the file has to arrive at boot. The volume-initialization script is the earliest hook available, and it already runs as root before any component starts. The bundle that declares the tokenizer is a separate requirement. A missing tokenizer file fails silently, leaving no tokenizer. A missing declaration leaves tie-cloud-app with no tokenizer and no fallback. Both failures look identical, so check the file first and the declaration second. The identifier has to be one the build hardcodes; the bundle in Assemble the overlay declares one that the verified build accepts. Assemble the overlay Four files come out of the base image. Read each one out of the image rather than from a reference copy, so they match the build you’re patching: % mkdir -p orig overlays assets causal-lm % for f in /mc/runc/config.afm-runtime.json \ /etc/systemd/system/afm-runtime.service \ /usr/bin/megacontainer-init-volumes \ /mc/eh-allowlists/components/afm-runtime.json; do container run --rm --platform linux/amd64 "$MCIMAGE" cat "$f" \ > "orig/$(basename $f)" done Point afm-runtime at the v11_pro repository, load causal_lm alone, select the inference server explicitly, and mount the no-op model over the shipped one. The container launcher reads config.afm-runtime.json at boot and merges it into the component’s runtime configuration, so you change a component’s environment and mounts by editing this file: % REPO=/opt/afm_runtime/triton_repos/v11_pro % jq --arg repo "$REPO" ' .process.env += ["TRITON_MODEL_REPOSITORY=" + $repo, "TRITON_LOAD_MODELS=causal_lm", "SERVER_TYPE=triton"] | .mounts += [{destination: ($repo + "/causal_lm/1"), source: "/mc/supports/noop-causal-lm", type: "bind", options: ["bind", "ro"]}]' \ orig/config.afm-runtime.json > overlays/config.afm-runtime.json Setting the repository explicitly also skips the manifest lookup the entry point otherwise does in /weights, so you need no model manifest and no version variable. Drop the exporter from the unit’s dependencies: % sed 's/ tie-model-owner\.service//' orig/afm-runtime.service \ > overlays/afm-runtime.service Append the commands that stage the tokenizer to the volume-initialization script, then put the tokenizer and the bundle that declares it in assets/: % cp orig/megacontainer-init-volumes overlays/megacontainer-init-volumes % cat >> overlays/megacontainer-init-volumes <<'SH' cp /mc/supports/noop-assets/tokenizer.model /weights/tokenizer.model cp /mc/supports/noop-assets/resource_bundles_v2.json /weights/resource_bundles_v2.json chown 1011:1011 /weights/tokenizer.model /weights/resource_bundles_v2.json SH % cp /path/to/gemma/tokenizer.model assets/tokenizer.model % cat > assets/resource_bundles_v2.json <<'JSON' {"resource_bundles": [{ "identifier": "com.apple.fm.language.instruct_server_v2.pcc_test_pro#cv10", "tokenizer": "com.apple.fm.language.instruct_server_v2.tokenizer_pro.generic"}]} JSON 1011 is aea-stream-downloader, the component that populates /weights on a node whose configuration downloads model weights; the stock script this appends to already modifies the directory’s ownership, and matching it keeps the ownership consistent. Both readers of these files — afm-runtime and tie-cloud-app — mount /weights read-only, so they need only read access, which cp from a root-owned source gives. Update the execution allow list. The path already has an entry, because you are replacing a file the image ships, so only the digest changes: % jq --arg key "$REPO/causal_lm/1/model.py" \ --arg digest "sha256:$(shasum -a 256 causal-lm/model.py | cut -d' ' -f1)" \ '.scriptEntries[$key] = $digest' \ orig/afm-runtime.json > overlays/afm-runtime.json Then write the Dockerfile. The EXPOSE and VOLUME lines restate what the base image declares, which Restore dropped image metadata explains and shows how to read; the values below are the ones the verified build declared: ARG MC FROM ${MC} EXPOSE 4442 EXPOSE 8080 EXPOSE 13987 EXPOSE 13999 VOLUME ["/var/lib/containers/overlays", "/var/lib/containers/run"] COPY causal-lm /mc/supports/noop-causal-lm COPY assets /mc/supports/noop-assets COPY overlays/config.afm-runtime.json /mc/runc/config.afm-runtime.json COPY overlays/afm-runtime.service /etc/systemd/system/afm-runtime.service COPY overlays/afm-runtime.json /mc/eh-allowlists/components/afm-runtime.json COPY --chmod=755 overlays/megacontainer-init-volumes /usr/bin/megacontainer-init-volumes RUN rm -f /etc/systemd/system/multi-user.target.wants/gpuhealthd.service \ /etc/systemd/system/multi-user.target.wants/tie-model-owner.service \ /etc/systemd/system/multi-user.target.wants/tie-model-owner.path \ /etc/systemd/system/multi-user.target.wants/tie-controllerd.service \ /etc/systemd/system/multi-user.target.wants/tie-controllerd.path Build it, tagged as the reference you intend to deploy, then Push and deploy: % container build --platform linux/amd64 --build-arg MC="$MCIMAGE" \ -t "$DESTIMAGE" . Bypass the external workload controller cloudboardd doesn’t serve requests until a workload controller registers a workload with it. On a fleet node the control plane does that; on an instance you create yourself there is no control plane, so cloudboardd waits indefinitely and the node never answers. The workload image is intact in this state; the symptom is silence rather than an explicit error. Set workload.controller.skipexternal to bypass it. cloudboardd then ignores the registerWorkload and setState calls it would otherwise wait for, and reads its workload configuration from local keys instead. The key is unattested, so it belongs in the unmeasured configuration override rather than in the image: % OVERRIDE='{"checksum": "override-1", "values": {"preferences": [ {"applicationId": "com.apple.cloudboardd", "key": "workload", "value": {"controller": {"skipexternal": true}}}]}}' Pass the result as tee-env-PCC_UNMEASURED_CONFIG_OVERRIDE in the metadata below. The defaults cloudboardd falls back to are a service name of pcc-default-service and a concurrency of one request; the service name is deliberately not one the Node selection accepts, which costs nothing here because requests reach this instance directly rather than through the gateway. Launch an instance without a GPU A node with GPUs attached comes from a reservation, and the machine type it uses, A4 as of this writing, is scarce. The image you just built needs no GPU, so you can create an instance for it yourself, in a project you own, on a machine type that is widely available. c3-standard-192-metal is the one this section uses, and gcloud alone launches it. Three things about that machine type differ from a node with GPUs attached, and each narrows what the instance can tell you: The host, the Workload VM, and most of the workload container remain, which is enough to exercise attestation, the encrypted request path, configuration delivery, the execution allow list, and telemetry egress. The machine type also constrains the host image and four metadata keys: One create call supplies everything the instance needs: the host image to boot ($BOOT_IMAGE) as the boot disk, and, in instance metadata, the Workload VM’s memory size, the workload container reference you pushed ($DESTIMAGE), the unmeasured configuration override ($OVERRIDE), and the capabilities, mounts, and shared-memory size the workload needs. The launcher reads the tee- prefixed keys; the host reads the rest before it starts the Workload VM. Two of the metadata keys have effects documented elsewhere: tee-image-reference is the same key that Replace the workload container updates, and tee-debug-ssh-enable opens the hop that Reach the host uses. The ^~^ prefix on the metadata string makes ~ the separator between entries, because the values themselves contain commas: % CAPS='["CAP_SYS_ADMIN", "CAP_NET_ADMIN", "CAP_MKNOD", "CAP_SETUID", "CAP_SETGID", "CAP_CHOWN", "CAP_FOWNER", "CAP_DAC_OVERRIDE", "CAP_DAC_READ_SEARCH", "CAP_SETPCAP", "CAP_SYS_CHROOT", "CAP_SYS_PTRACE", "CAP_KILL", "CAP_AUDIT_WRITE", "CAP_SYS_RESOURCE", "CAP_FSETID", "CAP_NET_BIND_SERVICE", "CAP_NET_RAW", "CAP_SETFCAP", "CAP_BPF", "CAP_PERFMON"]' % MOUNTS='type=tmpfs,source=tmpfs,destination=/var/run/cache,size=32000000000;type=tmpfs,source=tmpfs,destination=/var/lib/containers/run,size=32000000000;type=tmpfs,source=tmpfs,destination=/var/lib/containers/overlays,size=32000000000;type=tmpfs,source=tmpfs,destination=/weights,size=32000000000;type=tmpfs,source=tmpfs,destination=/private/var/MLModels/LLM/Prompts,size=32000000000' % FLEET_NODE_ID="${INSTANCE_NAME}" % METADATA="^~^\ guest-memory-size-gib=256~\ fleet-node-id=${FLEET_NODE_ID}~\ tee-image-reference=${DESTIMAGE}~\ tee-disable-gca-refresh=true~\ tee-container-log-redirect=serial~\ tee-cgroup-ns=true~\ tee-dev-shm-size-kb=1000000~\ tee-debug-ssh-enable=true~\ tee-env-PCC_VARIANT=afm-vre~\ tee-env-PCC_UNMEASURED_CONFIG_OVERRIDE=${OVERRIDE}~\ tee-added-capabilities=${CAPS}~\ tee-mount=${MOUNTS}" % gcloud compute instances create "${INSTANCE_NAME}" \ --machine-type=c3-standard-192-metal \ --image="$BOOT_IMAGE" \ --boot-disk-size=32GB \ --boot-disk-type=hyperdisk-balanced \ --maintenance-policy=TERMINATE \ --scopes=cloud-platform \ --metadata="$METADATA" The launcher validates each size in $MOUNTS on its own against free memory in the Workload VM, and refuses to start the workload when any single entry exceeds it. It doesn’t check the sum, and a size buys no overcommit, so the much larger values a node with GPUs attached uses don’t carry over. tee-dev-shm-size-kb goes through the same check. An entry that is too large stops the boot before any component starts, and the error message gives both numbers: got a /dev/shm size (357421875) larger than free memory (92397872) kB Take the free-memory figure from that message rather than deriving it from the machine’s memory or from guest-memory-size-gib, because it is smaller than either. The figure also moves between boots, so leave headroom rather than sizing to the number you read. The machine type fixes three of the flags rather than leaving them to you: hyperdisk-balanced is the only boot-disk type it offers, a machine of this type cannot live-migrate, and the launcher needs cloud-platform scope to fetch an attestation token. The network path to the instance is yours to configure, because the project is yours. The tunnel that Reach an instance describes works here, and so does any other path you set up. Expect the same wait either way: the instance reports RUNNING well before the workload answers. The service account is yours to choose for the same reason, and it needs three grants: read on the registry holding $DESTIMAGE, roles/confidentialcomputing.workloadUser on the project so the launcher can fetch its attestation token, and read on any bucket a configuration override names. A project owner’s default compute service account usually already carries them; an account in a project with restrictive policy does not, and the launcher fails before it pulls the workload container. Confirm the result The afm-runtime container entry point reports the repository and server it resolved, and the server reports which models it loaded: % journalctl -u afm-runtime | grep -E "Server Type|Model Repository|successfully loaded" Server Type: triton (--server-type / SERVER_TYPE) Model Repository: /opt/afm_runtime/triton_repos/v11_pro "successfully loaded 'causal_lm'" Some log lines look like failures but are not: The absence of image_tokenizer, which TRITON_LOAD_MODELS=causal_lm leaves unloaded. Cannot get CUDA device count, GPU metrics will not be available. No model manifest found path=/weights/manifest.json. Huge-page KV cache buffer allocation failed; falling back to normal pages. Confirm the stand-in itself is what loaded, and that its digest matches the allow list entry you wrote. Read it from inside the component’s container: runc applies the bind mount in the container’s own mount namespace, so the same path read from the workload container shows the shipped file instead. % runc exec "$(runc list -q | grep '^afm-runtime-' | head -1)" \ sha256sum /opt/afm_runtime/triton_repos/v11_pro/causal_lm/1/model.py % jq -r '.scriptEntries["/opt/afm_runtime/triton_repos/v11_pro/causal_lm/1/model.py"]' \ /mc/eh-allowlists/components/afm-runtime.json Then Issue inference requests. Read the shape of the response rather than a token count: tie-cloud-app asks for the context length minus the prompt, so a complete stream runs to a six-figure number of token responses, and the exact figure moves with the prompt and the tokenizer. The request passes when the stream ends with one final response and a summary of ok. A single token response means generation stopped at the first token; read its text for the reason. Understand what these changes don’t do The node attests to a different measurement than the published build, because every change above is inside the image. That is the intended behavior, and the same property that Modify measured configuration relies on. It also means a result from this instance is not a result about the published image. Inspect an attestation bundle. Where a finding does not depend on the absence of a GPU, reproduce it against the unmodified image before you report it. A machine type with no GPU narrows the attestation as well, and that limit comes from the machine rather than from the image. With no GPU present the attestation carries no GPU measurements, and with no TPM exposed the host contributes none either. An attestation from this instance therefore covers a smaller stack than one from a node with GPUs attached. Reproduce any finding about attestation coverage of the GPU or the host on a node booting the hardened host image with GPUs attached before you report it. A standalone instance also carries no ensemble identity. cb-attestationd reads that identity from the optional FLEET_NODE_ID configuration key, which a control plane supplies. That key is distinct from the mandatory fleet-node-id instance-metadata attribute that cpagent reads. With the key unset, attestation still succeeds: cb-attestationd leaves the ensemble field out of the attestation AppData and populates the boot identifier as usual. When you compare bundles from this instance with a fleet node’s, treat the ensemble field as unset. Two further limits follow from the overlay rather than from the node. The stand-in serializes requests, and bypassing the workload controller leaves cloudboardd with a local concurrency setting of one request, so this instance doesn’t exercise request concurrency on either count. The staged tokenizer splits control tokens and mismatches the stop-token IDs tie-cloud-app hardcodes, so the token stream is not evidence about tokenization.