Visual Lookup

Overview

Visual Lookup (VLU) enables users to identify visual entities — such as pets, landmarks, art, and books — by matching image embeddings against a database of known entities. Because VLU queries reveal information about content a user views, we designed the system to help protect query privacy by pairing PCC with private information retrieval.

VLU workflow

VLUAgent is a PCC Agent workflows that performs visual search using the following steps:

• Receives an embedding and metadata from the user’s device

• Retrieves a target cluster from the visual entity database using Private Information Retrieval (PIR)

• Applies custom ranking logic within the cluster

• Returns candidate documents with resolved metadata to the user’s device

Visual entity database

The visual entity database contains embeddings and metadata for known entities. Its cluster index is hosted on a server outside the PCC High-security inference, so VLU relies on PIR to prevent the database server from learning which entries a query accesses.

The database is organized into clusters to enable an efficient two-stage search: first selecting the index of the most relevant cluster using the codebook and retrieving it using PIR, then identifying the best match within that cluster. This clustering approach reduces the search space for each query and lowers the computational cost of PIR operations.

The database consists of three data structures:

Cluster codebook

Maps cluster index to centroids. cb_jobhelper uses CipherMLSE with the codebook to determine which cluster is most relevant for a given embedding.

Cluster index

Stores a cluster’s entries, the embeddings and their metadata. A PIR database is generated for the cluster index and hosted by a PIR server, so that PIR queries can access individual clusters without revealing the target cluster to the server.

Stash

Some clusters exceed the size constraints imposed by PIR hint storage. To handle these clusters, the server adds them to a stash, a data structure that holds key-value pairs. During request processing, if the target cluster is in the stash, cb_jobhelper takes the cluster contents from the stash instead of from the PIR response. To prevent timing side channels, the system issues PIR queries regardless of whether the stash contains the requested data.

Cloud asset distribution delivers both the stash and the codebook to PCC nodes.

An index can also ship to the node in full. When the delivered asset includes an offline variant of an index, CipherMLSE answers lookups against it from node-local storage without a PIR query. VLUAgent searches that variant alongside the PIR-backed indexes for book and media lookups, so a result in those domains can come from an index the node holds in full.

Request flow

VLU query processing proceeds as follows:

1. The user’s device generates an image embedding on-device. Because the embedding is generated locally, the raw image does not leave the device.

2. The user’s device uses the PCC client to send this embedding to the VLUAgent workflow.

3. VLUAgent makes a Delegated tool calls with the embedding to a PCCAgentWorkerApp running on another PCC node.

4. PCCAgentWorkerApp forwards the embedding to its associated cb_jobhelper, which uses the CipherMLSE framework to find the cluster centroid in the codebook nearest to the embedding. This determines which cluster is most likely to contain matching entities.

5. cb_jobhelper generates and submits a PIR query for the index of the relevant cluster, following the Request flow.

6. cb_jobhelper returns the resulting cluster (from either the decrypted PIR response or the stash) to PCCAgentWorkerApp, which returns it to VLUAgent. Both PCCAgentWorkerApp and cb_jobhelper then terminate.

7. VLUAgent re-ranks the results within the cluster to find the nearest or best-matching entries.

8. VLUAgent performs additional delegated tool calls to resolve the metadata for the best-matching entries.

9. VLUAgent attaches metadata (such as titles) to the best-matching entries and returns them to the user’s device.

For additional information about the cryptographic operations performed, see Private Information Retrieval.

CipherMLSE framework

The CipherMLSE framework provides VLU-specific operations for embedding-based search:

Cluster selection

CipherMLSE implements nearest-neighbor search to determine which cluster is most likely to contain matches for the input embedding. This computation uses the cluster codebook to identify the target cluster index.

Query formatting

Once cb_jobhelper determines the cluster index, CipherMLSE prepares the encrypted PIR query for that index, following the Request flow.

Response processing

After the PIR server returns the encrypted response, CipherMLSE decrypts it to obtain the cluster contents.

Re-ranking

VLUAgent performs re-ranking within the retrieved cluster by computing similarity metrics between the input embedding and the embeddings in the cluster.

Security properties

Visual Lookup is designed to provide security and privacy properties specific to visual search:

Embedding privacy

The device embedding stays within the PCC trust boundary, where cb_jobhelper uses it to select a cluster on the worker node and VLUAgent uses it to re-rank within that cluster on the agent node. Security properties help ensure that neither the PIR server nor the database operator can determine which cluster a query targets, preventing inference of the user’s embedding.

Stash privacy

The stash mechanism enables PIR for large clusters that would otherwise have prohibitive hint sizes, without compromising privacy. The system delivers the stash to PCC nodes as a Cloud asset distribution, so the use of stash data does not require communication with any external server during query processing. To prevent timing-based side channels, PIR queries run regardless of whether the target cluster is found in the stash. This approach helps ensure that a network adversary cannot infer whether a request was served from the stash by observing traffic to and from the PIR server.