Embedding Generation

Overview

PCC Agent can generate embedding vectors on a PCC node from text and decoded video frames through the MediaAnalysis frameworks. This capability is exposed through the PCC Agent workflows, a workflow that returns embeddings to its caller. It is also exposed as a tool that other PCC Agent workflows invoke to embed content as part of a larger task.

Embedding tool

Embedding generation is exposed to workflows as the EmbeddingTool, a Delegated tool calls provided by PCC Agent. A workflow invokes the EmbeddingTool with a single request that carries one or more content entries of the same kind, for example, a text string, a batch of text strings, or a sequence of video frames, and receives one embedding vector per entry.

The EmbeddingTool runs as a delegated tool call: for each embedding request, the agent issues a one-shot call to a worker node, which generates the embedding in its own per-request PCCAgentWorkerApp address space and returns the result to the workflow. A worker invocation handles a single embedding request, so even when one workflow makes several embedding calls, each runs in a separate worker process with a lifetime bounded by that request.

On the worker node, the embedding tool validates the request, resolves the requested model version against the set of models the worker advertises, and dispatches to one of two execution paths:

Text embedding

A single string or a batch of strings is passed to MADEmbeddingService through the MADTextEmbeddingRequest API, which returns one embedding vector per input.

Video embedding

Each video frame is submitted to MADEmbeddingService through the MADVideoEmbeddingRequest API, producing one embedding per frame, returned alongside the frame’s timestamp.

Pixel data, such as the frames of a video, is carried on the wire as an uncompressed pixel buffer. Before submitting that data to MediaAnalysis, the tool validates it: it accepts only a fixed set of pixel formats and bounds each buffer’s dimensions, stride, and size, rejecting any input outside those limits.

Image embedding is not currently supported; the embedding tool rejects any request to embed an image.

MediaAnalysis frameworks

Embedding inference is implemented by MediaAnalysis through three frameworks:

EmbeddingCore

Implements the embedding models and encoders.

MediaAnalysisCore

Provides the image and video-frame processing the encoders build on.

MADEmbeddingService

Exposes the interface that the worker calls, and is built on the other two.

PCC links these frameworks into the worker and runs embedding in the worker’s own process rather than calling out to a separate daemon, so embedding generation inherits the sandbox, entitlements, and per-request lifetime of PCCAgentWorkerApp. The tool passes text inputs directly to the model.

Model delivery

The embedding model is delivered as a Codeless cryptexes. MediaAnalysis parses the weights through Core ML and BNNS-IR back ends, whose input formats are complex enough that we don’t treat parsing them as inherently safe. As with other published codeless cryptexes, the embedding model is therefore held to PCC’s Verifiable Transparency guarantees and published for independent inspection using the Virtual Research Environment tools.