Contents

Providing contextual cues to Apple Intelligence and Siri

Annotate your interface with app entities to offer contextual information about your app’s onscreen content.

Overview

Your app’s interface reflects what a person is doing, and what data they’re viewing or changing. Although your app’s interface contains data, the system can’t use that data directly because it’s private to your app. However, you can annotate views and other content with app entities to give the system the context it needs to improve other types of interactions. For example, when someone refers to parts of your app’s content during a Siri conversation, Apple Intelligence can use the contextual information you provide to understand what the person meant.

Apply a domain-specific schema to your custom type

The App Intents framework defines domains for different types of content, and schemas for the app intents and app entities related to those domains. Each schema specifies the expected format for app intents and entities, including any expected parameters or properties. If your content matches one of the existing schemas, apply that schema to your type and implement the required content.

To adopt one of the predefined schemas for an app entity, type the @AppEntity macro before your AppEntity type declaration and use code completion to specify the schema you want. Add properties to your app entity to match the schema you selected. For properties that are part of the schema definition, the system infers the @Property macro automatically, so you don’t need to add it. For more details on how to create and implement entities, see Defining app entities for your custom data types.

Support conversions between your app entities and equivalent system types

Many apps deal with similar types of content, and might define custom app entities to represent that content. Although you can use the Transferable protocol to provide alternate representations of your content, there are limitations to how other apps can use that content. The protocol supports the creation of files and binary data types, which limits what apps can do with the content. For example, Maps can’t use a picture of a landmark to generate directions to that landmark. For locations, names, and contacts, you can use the IntentValueRepresentation type to provide a version of your data that other apps can use directly.

If your entity offers the same information found in an IntentPerson, PlaceDescriptor, or PersonNameComponents type, implement the transferRepresentation property and add an IntentValueRepresentation as one of the representations. A representation of this type supports both importing and exporting your type. The following example shows an entity that manages contact information. The code provides the contact information in an IntentPerson structure so that other apps can incorporate that contact information directly.

struct ContactEntity: AppEntity, Transferable {
    static var typeDisplayRepresentation: TypeDisplayRepresentation = "Contact"
    
    var id: String
    @Property var name: String
    @Property var email: String
    
    var displayRepresentation: DisplayRepresentation {
        .init(title: "\(name)")
    }
    
    static var defaultQuery = ContactEntityQuery()
    
    // Bidirectional conversion with IntentPerson.
    static var transferRepresentation: some TransferRepresentation {
        IntentValueRepresentation(
            exporting: { contact in
                IntentPerson(
                    identifier: .applicationDefined(contact.id),
                    name: .displayName(contact.name),
                    handle: .init(emailAddress: contact.email)
                )
            },
            importing: { person in
                guard case let .applicationDefined(id) = person.identifier?.value,
                      let handle = person.handle else {
                    throw ConversionError.missingData
                }
                return ContactEntity(
                    id: id,
                    name: person.name.displayString,
                    email: handle.value
                )
            }
        )
    }
    
    enum ConversionError: Error {
        case missingData
    }

For more information about making your types transferable, see the Core Transferable framework.

Associate entities with the views in your interface

Your app’s interface reflects the things a person is doing and what data they’re viewing or changing, but that information remains private to your app unless you tell the system about it.

To share your app’s content with the system, attach app entities to the views you use to show content:

If you don’t have a one-to-one mapping between an app entity and one of your views, add entities using the AppEntityUIElement type. You might use this approach if your app draws all of its content in a single view or places content in CALayer. With an entity UI element, you use a closure to supply the app entities for your view on demand. In SwiftUI, provide this closure using the appEntityUIElements(_:) modifier. In UIKit and AppKit, assign this closure to your view’s appEntityUIElementProvider property.

The following examples show how to use a UI element closure to specify entities for SwiftUI and UIKit views. The view in the example draws custom sticky notes, which the person can place anywhere in the view. The closure uses the system-provided context object to determine whether to return the selected notes or the notes within the specified rectangle. For each note that matches the requested criteria, it creates an AppEntityUIElement type with the entity information and returns it to the system.

Associating app entities with your views is the preferred way to make your content known to the system, but other options are available for apps that need to support earlier operating system versions. User activity objects have a appEntityIdentifier property, which you can use to specify data for an action.

In UIKit and AppKit, you can assign user activity objects to views and responders using their userActivity property. In SwiftUI, you can create and configure an activity object in one of your views using the userActivity(_:element:_:) modifier.

Associate entities with other types of content

If a system feature displays app-specific data, check whether you can provide an app entity with that data. Some system types support the AppEntityAnnotatable protocol, which gives you a way to associate an app entity with that type. For example:

See Also

Onscreen context