Making your indexed content available to Foundation Models

Make the content you index for Spotlight available to Foundation models to help generate responses to prompts.

Overview

The Foundation Models framework provides access to language models, which you can use to implement intelligent features in your apps. Language models answer questions on a wide variety of subjects, but they don’t necessarily know how to answer specific questions about your app’s content. To make your data available to the model, add the Spotlight search tool to the configuration of your Foundation Model sessions.

The Spotlight search tool gives models a way to search your app’s content and use the results to answer prompts. When a model requires your app’s data, it uses the search tool to run queries against your app’s Spotlight index or indexed files. The model uses the results as additional context when answering prompts that involve your content.

Configure a model to use the Spotlight search tool

The Foundation Models framework supports the inclusion of tools in a session to handle specific tasks. The SpotlightSearchTool type implements the framework’s protocols for tools and gives models the ability to query your app’s content. When initializing a Foundation Models session, include the Spotlight search tool and configure it with app-specific search options. The following example creates a default instance of the tool, uses it to initialize a session, and generates a response to a prompt:

import CoreSpotlight
import FoundationModels

let tool = SpotlightSearchTool()

let session = LanguageModelSession(tools: [tool])
let response = try await session.respond(to: "Find my notes about the project deadline.")

The default configuration of SpotlightSearchTool performs queries against your app’s Spotlight index using all available search techniques. When creating the tool, you can provide custom guidance on how to search your content to improve the efficiency of queries. You can customize the tool’s configuration in the following ways:

  • Specify where to find your app’s content. You can tell the tool to search your app’s Spotlight index or search directories containing your app’s files. You can also search multiple sources and combine the results.

  • Tell the tool to fetch specific attributes of each search result and provide them to the model. The default tool configuration returns minimal information for each item. Fetching attributes during a search provides the model with additional information it can use to answer prompts.

  • Optimize data retrieval operations. You can tell the tool to retrieve attributes related to someone’s communications, documents, media, or other types of data. The tool packages this information in a compact format suitable for on-device models or models with smaller context windows.

  • Offer guidance on how to search your content. The Spotlight search tool has many ways to perform searches, but some of those techniques might not apply to your content. For example, if you only search for textual items, you don’t need to consider searches involving dates or numerical values. Eliminate search options you don’t need to reduce the amount of content the tool sends to the model.

Specify where to find your app’s content

At configuration time, you need to tell the Spotlight search tool where to look for your app’s content. You provide that information using a search source, which is one of the following types:

In addition to telling your app where to search, a search source provides details about how to perform searches of that source. You can configure each source with the maximum number of results to return during a query. You can also specify which attributes you want to retrieve for search results. For the CoreSpotlightSource type, you can also specify a delegate object to recreate attributes that the tool can’t retrieve from the index. Configuring these values helps the tool deliver better information for each query.

You can configure the Spotlight search tool with multiple sources, and search your app’s Spotlight index, multiple directories, or both. During a query, the tool searches each source separately and then combines the results before delivering them to the model.

Make searches more efficient by optimizing data retrieval

At configuration time, you can provide guidance to help the model optimize the query it performs using the Spotlight search tool. The search tool supports a variety of search techniques and content types, and you use guidance to limit the options available to the model. Eliminating options that aren’t relevant for your content reduces the amount of information the search tool sends back to the model, freeing up context space for other types of content.

To specify your guidance options, create the SpotlightSearchTool.Configuration structure and add your guidance to the guide property. When configuring guidance, you can choose from the following options:

If you’re using a model with severe token constraints, another way to improve efficiency is to apply the SpotlightSearchTool.FormatLevel.compact configuration option. When you apply this option to the format property of your guide configuration, the tool outputs results in a compact format. You might choose this option if you’re using a model with a small context window that can’t handle conversations with large amounts of data. You might also use this approach if you anticipate a long conversation and need to save room for additional contextual data later.

Help the model resolve references to people

People inherently understand the personal pronouns other people use during conversations, but those same pronouns present difficulties for tools and code. If your searchable items contain metadata relating to people, such as author or recipient data, provide a contact resolver to help the model resolve first-person pronouns like “I” and “me”. A contact resolver provides information about the person using your app such as the person’s name, email, or phone details. The model can match this information against the data it encounters in item metadata to resolve first-person references. The following example shows how to add a custom contact resolver to the Spotlight search tool:

let tool = SpotlightSearchTool(configuration: .init(
    contactResolver: MyContactResolver()
))
let session = LanguageModelSession(tools: [tool])

// "I" is resolved to the user's identity for matching against item authors/senders.
let response = try await session.respond(to: "Show me the documents I shared last week.")

To implement a contact resolver, create a type that adopts the ContactResolver protocol. Use your custom type to return a ResolvedContact structure with information about the person who owns your app’s data. Fill in the properties of the structure with relevant information about the person who uses your app. For example, a communications app might include the person’s name and the phone number or email associated with their account. The search tool passes this information along to the model to help it reason about references to the person.

Customize how the tool determines search results

When processing complex prompts, a model might run multiple queries to get the results it needs. For each query, the model builds a pipeline of work, where each stage of the pipeline is a specific task to run on the data. For example, one stage might retrieve items, another one might count the items, and a third one might assign relevance scores to each item. The Spotlight search tool defines pipeline stages for many common tasks, but you can add custom stages to apply app-specific information to the data in the pipeline.

The model builds tool pipelines dynamically and can run stages in any order, so any stages you create need to run indendently of other stages. When you define a custom stage, you specify the type of input data you want to receive and the type of output data your stage generates. For example, a stage that creates relevance scores for items might take CSSearchableItem objects as input and generate ScoredSearchableItem objects as outputs. The inputs and outputs, plus other information in your custom stage, govern when a model might add that stage to a pipeline.

To define a custom stage, create a structure that conforms to the CustomStage protocol. Use the protocol’s API to specify the inputs and outputs of your stage, and use the description property to provide the model with instructions on how to use your stage. Implement the execute method you support to transform the data you receive to the expected output data. The following example shows an implementation of this protocol that ranks each searchable item based on how recently the person viewed it.

struct RecencyBoostStage: CustomStage {
    static var name: String { "recency_boost" }
    static var description: String { "Boosts recently modified items in the ranking." }
    static var inputTypes: [SearchPipelineDataType] { [.items] }
    static var outputTypes: [SearchPipelineDataType] { [.scoredItems] }

    var recencyWeight: Double

    func execute(items: [CSSearchableItem]) async throws -> SearchPipelineData {
        let now = Date()
        let scored = items.map { item -> ScoredSearchableItem in
            let age = now.timeIntervalSince(item.attributeSet.contentModificationDate ?? .distantPast)
            let recencyScore = max(0, 1.0 - (age / (30 * 86400))) // decay over 30 days
            return ScoredSearchableItem(item: item, score: recencyScore * recencyWeight)
        }
        return .scoredItems(scored)
    }
}

To simplify the creation of your custom stage, extend the CustomStage protocol and add a static function to create your custom structure. You can use this function later to create instances of your stage using dot syntax. The following example defines the recencyBoost function that configures and returns a new instance of the RecencyBoostStage structure.

extension CustomStage where Self == RecencyBoostStage {
    static func recencyBoost(weight: Double = 0.3) -> Self {
        RecencyBoostStage(recencyWeight: weight)
    }
}

To add your custom stage to the Spotlight search tool, specify it at configuration time as shown in the following example. The code uses dot syntax to take advantage of the custom function from the previous code listing.

let tool = SpotlightSearchTool(configuration: .init(
    sources: [.coreSpotlight],
    customStages: [.recencyBoost(), .recencyBoost(weight: 0.5)]
))

Examine results while a search runs

While the Spotlight search tool runs, your app can monitor the search results and any search-related information the tool generates. You might do so to debug your code, to display the search results in your app, or to monitor specific metrics during the search process. To receive these results, set up a task to monitor the asynchronous sequence in the searchResults property of SpotlightSearchTool. This property delivers a series of SpotlightSearchTool.SearchReply structures to your app, which you use to retrieve information.

During prompt resolution, the model might run multiple queries, and each query might require multiple stages to generate the needed data. The queryToken and stageToken properties of each SpotlightSearchTool.SearchReply structure help you correlate each instance to a particular query and stage. Use this information to determine how to organize this information in your app. For example, you might group the results from each query or stage, or refresh the list of results each time a new query starts.

When processing results, use the content property of the SpotlightSearchTool.SearchReply structure to determine what information the type contains. The following example creates a task and uses a for await loop to receive the reply structures asynchronously. With each new structure, the code updates different parts of its interface to show the data. The code shows a progress indicator while it waits for more data from the same query.

let tool = SpotlightSearchTool()
let session = LanguageModelSession(tools: [tool])

// Start consuming results before the model responds.
Task {
    var currentToken: SpotlightSearchTool.SearchReply.QueryToken?
    for await reply in tool.searchResults {
        if reply.queryToken != currentToken {
            // New query — start a new display section.
            currentToken = reply.queryToken
        }

        switch reply.content {
        case .items(let items):
            displayResultsList(label: reply.label, items: items)

        case .count(let n, let header):
            displayCount(n, header: header ?? reply.label)

        case .table(let table):
            displayTable(table)

        case .statistic(let name, let value, let header):
            displayMetric(name: name, value: value, header: header ?? reply.label)

        case .text(let body, let header):
            displayTextBlock(body, header: header ?? reply.label)
        }

        showProgressIndicator(reply.status == .partial)
    }
}

let response = try await session.respond(to: "Show me recent emails from Shelly.”)