HomeKit Secure Video Analysis

Overview

This workflow brings cloud analysis to a user’s HomeKit Secure Video (HKSV) clips; its HKSVProcessingAgent is a PCC Agent workflow that runs video analysis on behalf of the user’s home. It produces natural-language video descriptions and embeddings that the Home app then uses to summarize clip activity, run free-form semantic search across the user’s video library, and coalesce notifications. The user’s home hub triggers processing: it receives the clip from the HomeKit camera and submits the encrypted video fragments to PCC inside the request body for decoding and analysis. PCC processes only the encrypted fragments the home hub submits in the request; it does not retrieve clips or fragments from any other store.

Workflow inputs

The home hub initiates processing by sending a request to the HKSVProcessingAgent workflow. The request carries three kinds of input:

Clip and fragments

A clipID identifying the video clip and an array of HKSVFragment values, each containing the raw bytes of a video fragment (H.264 or H.265) along with the fragment number and a presentation start time and duration. A single request operates on one clip composed of multiple fragments.

Spatiotemporal metadata

Per-fragment face metadata produced on the home hub, including bounding boxes, person labels, person UUIDs, and frame-relative timestamps. The home hub populates this metadata before submitting the request; PCC does not perform face detection. PCC uses this metadata only to describe the current clip and retains no analysis state between requests.

Prior analysis results

For each request, the home hub re-sends the embeddings and video descriptions it previously received from PCC for nearby clips. The workflow’s gating step uses these to skip generating a description when the new clip is sufficiently similar to one already analyzed. Because the workflow does not persist analysis state across requests, the home hub keeps prior results and re-sends them with each request.

Processing options carried in the request select an operating mode (such as EmbeddingOnly to skip generating descriptions entirely, or a histogram-reuse mode driven by a HistogramReuseClipID that refers to a prior clip whose analysis the device reuses rather than recomputes), a similarity threshold for the gating step, and a client locale that selects the description prompt language.

Workflow execution

Following the PCC Agent workflows, HKSVProcessingAgent runs a static, four-stage pipeline on an PCC Agent, issuing Delegated tool calls to other PCC nodes for three of those stages:

Decode

The camera fragments are untrusted third-party input, so the workflow does not decode them itself. The workflow uses Media Decoding workers to decode each fragment and receives back only pixel buffers.

Embedding

The workflow flattens the decoded frames for each fragment into an embedding request and invokes the Embedding tool.

Gating

The workflow compares the new embeddings against the prior-result embeddings re-sent by the home hub. If the device requested embedding-only processing, if it supplied a HistogramReuseClipID referring to a prior clip with video descriptions that can be reused, or if the gating step found a sufficiently similar prior clip, the workflow returns the new embeddings together with the prior video descriptions and skips generating descriptions entirely.

Description generation

The workflow submits a single-turn chat prompt to an inference node. The model is prompted to emit a JSON object with a fixed set of fields, and the workflow parses each field into a typed HKSVCaption value before returning.

After generating the descriptions, the workflow assembles an HKSVResult that contains the embeddings, the video descriptions, and per-stage timing metadata, and returns this result to the home hub.