Visual Generation

PCC runs the Apple Diffusion Model to generate images.

Overview

Visual Generation runs the Apple Diffusion Model (ADM) on PCC to produce images for on-device features such as Image Playground and Photos editing. Running ADM on PCC lets these surfaces use a larger, higher-quality model than is feasible on device. Visual Generation is implemented as a PCC Agent workflows, VisualGenerationAgent, which orchestrates the diffusion pipeline together with content-safety guardrail models and a Media Decoding.

Request data

The client encodes the generation request as a VisualGenerationInferenceRequest that contains a GenerationRecipe describing the work to perform:

  • The text prompt

  • Generation directives such as image style or aspect ratio

  • Input images for the specific feature

Each input image is delivered as a HEIC bitstream. We chose HEIC, which uses HEVC, for visual fidelity and application flexibility; JPEG lacks support for modern image-container features such as HDR, high-bit-depth color, and transparency, and is generally less efficient than HEVC.

Before decoding begins, VisualGenerationAgent checks the dimensions each input image declares, per image and summed across the request, against fixed pixel-area limits and rejects a request that exceeds either limit, bounding the decode work an untrusted client can request. The Media Decoding bounds the decoded dimensions independently.

Because these HEIC bitstreams are untrusted, VisualGenerationAgent does not decode them in-process. It walks the GenerationRecipe for HEIC-bearing image slots and sends the bitstreams it finds to the dedicated media decode worker in one call, receiving back raw pixel surfaces with metadata such as pixel format, bytes-per-row, and color space. The decoded pixel buffers then replace the HEIC bitstreams in the in-memory GenerationRecipe during further processing.

VisualGenerationInferenceResponse contains a HEIC-encoded result image, an error, or an intermediate result when a single ADM inference server cannot complete the full inference.

Inference orchestration

A request runs up to three stages:

  1. When a request includes a prompt, an input guardrail screens it (along with any image being edited) and rewrites the prompt into an expanded form for the diffusion model. If the guardrail rejects the prompt, the request terminates with an error before diffusion runs. Operations that supply no prompt, such as inpainting, outpainting, spatial reframing, and up-resolution, proceed directly to diffusion.

  2. The ADM diffusion loop generates the image from the rewritten prompt.

  3. An output guardrail screens the generated image.

The input and output guardrails are safety models that operate like other PCC models: PCC delivers them in a Foundation model and adapter cryptexes and runs them on GPU through MetalLM. The input guardrail receives the prompt and, for editing workflows, the image being edited; the output guardrail receives the generated image alone. Each emits a categorical verdict, and VisualGenerationAgent includes the verdict in the response so the client can render an appropriate error.

The request’s inference identifier determines how these stages map onto inference nodes:

Single invocation

For most identifiers, all three stages run in one The Inference Engine invocation on a single inference ensemble.

Distributed

When the identifier ends in _large_image, the stages are split across separate inference engine instances on different inference nodes: one for the input guardrail and text encoder, one for the diffusion operation, and one for the output guardrail.