Working with generic spatial accessories

Use generic spatial accessories to track purpose-built devices in your visionOS app.

Overview

Specialized apps become more immersive when they respond to people using purpose-built devices, like medical instruments, steering rigs, or industrial tooling. ARKit recognizes these devices as generic spatial accessories, but they’re distinct from spatial controllers and styli, which have their own dedicated APIs.

Manufacturers create generic spatial accessories by following the Apple Accessories Guidelines for creating a spatial accessory. ARKit provides precise, low-latency tracking of these accessories in Apple Vision Pro across varied lighting conditions, and continues tracking orientation even when an accessory moves outside the field of view or becomes visually obscured. Game Controller also provides input and haptic feedback for these accessories.

Configure access to a reference accessory file

To support a generic spatial accessory, your app needs a .referenceaccessory file which describes the device’s physical characteristics. ARKit uses the metadata in this file to recognize and track the device in physical space.

When your app initializes an Accessory from a connected GCSpatialAccessory, ARKit resolves the corresponding .referenceaccessory file using the Uniform Type Identifier (UTI) registered by the system. If ARKit can’t find a matching file, initialization fails. For more information, see Uniform Type Identifiers.

If you’re the accessory manufacturer, bundle the .referenceaccessory file in your app and declare it under UTExportedTypeDeclarations in your Info.plist. This both enables your app to use the file and registers the type system-wide, so any other app installed on the device can also use the file as long as your app is present.

To access generic spatial accessories from other manufacturers, check with the manufacturer to see if they have a .referenceaccessory file available. If so, bundle it in your app and declare it under UTImportedTypeDeclarations in your Info.plist so your app works even when the manufacturer’s app isn’t installed. An imported declaration tells the system you depend on the type but don’t own it. If more than one app declares an imported type for the same identifier, the system resolves to one of them.

If your app includes a UTImportedTypeDeclarations entry and the manufacturer’s app with a matching UTExportedTypeDeclarations entry is also installed, ARKit always gives the exported declaration precedence.

To bundle a .referenceaccessory file, drag it into your Xcode project and click the checkbox next to your app target in the dialog that appears. Then add a UTExportedTypeDeclarations entry to your Info.plist if your app owns the type, or a UTImportedTypeDeclarations entry if you depend on a type defined by someone else.

<key>UTImportedTypeDeclarations</key>   <!-- Or `UTExportedTypeDeclarations` if your app defines the type. -->
<array>
    <dict>
        <key>UTTypeConformsTo</key>
        <array>
            <!-- Confirm with the manufacturer -->
            <string>com.apple.spatial-device</string>
        </array>
        <key>UTTypeIdentifier</key>
        <!-- Replace with the manufacturer's identifier -->
        <string>com.example.mycontroller</string>
        <key>UTTypeReferenceAccessoryFile</key>
        <!-- Filename of a `.referenceaccessory` bundled in your app's resources. -->
        <string>my_controller.referenceaccessory</string>
        <key>UTTypeDescription</key>
        <!-- Human-readable name -->
        <string>My Controller</string>
    </dict>
</array>

UTTypeConformsTo: The parent type in the UTI conformance hierarchy. Check with the manufacturer for the correct value; com.apple.spatial-device is a reasonable default for spatial accessories.
UTTypeIdentifier: The unique reverse-DNS identifier for this accessory type, provided by the manufacturer. Use the exact value of the uniformTypeId defined for the object class when training the .referenceaccessory in Create ML.
UTTypeReferenceAccessoryFile: The exact filename of the .referenceaccessory bundle included in your app.
UTTypeDescription: A human-readable name the system uses for display and debugging.

For more information on Uniform Type Identifiers, see Defining file and data types for your app.

If the manufacturer doesn’t make the .referenceaccessory file available for bundling, your app can instead rely on ARKit to find the file when the manufacturer’s app is installed. In that case, handle initialization failures gracefully. When init(device:) throws, display an error that directs people to install the required app.

do {
    let trackedAccessory = try await Accessory(device: accessory)
    // Use the tracked accessory.
} catch {
    logger.error("Failed to create accessory: \(error)")
    // Prompt the person to install the manufacturer's app.
}

Add the accessory tracking capability

To help protect people’s privacy, visionOS limits app access to spatial accessory data and other sensor data on Apple Vision Pro. Add the Accessory Tracking capability to your app’s target and provide a usage description that explains how your app uses spatial accessory data. People see that description when the system prompts for access to accessory-tracking data. For more information on app capabilities, see Adding capabilities to your app.

Obtain authorization to track accessories

To read an accessory’s transform, your app needs Accessory Tracking authorization. Monitor the authorization status so your app can enable or disable features that require the transform.

At startup, use an ARKitSession to query the current authorization status.

// Query initial authorization status.
let authorizationStatus = await arkitSession.queryAuthorization(for: [.accessoryTracking])[.accessoryTracking] ?? .notDetermined
// Use authorization status to initialize your app.

Observe authorization changes on the same session so your app can respond when people grant or revoke permission:

// Observe and respond to authorization changes.
for await event in arkitSession.events {
    switch event {
    case .authorizationChanged(.accessoryTracking, let status):
        // Handle changes to authorization status.
    default:
        break
    }
}

Discover connected accessories

Before your app can anchor content to an accessory, track it, or respond to its input, your app needs a reference to the GCSpatialAccessory instance for the connected accessory.

Query spatialAccessories for a list of connected accessories:

for accessory in GCSpatialAccessory.spatialAccessories {
    // Keep a reference to the accessory.
}

Listen for connect notifications to handle accessories that connect later:

for await notification in NotificationCenter.default.notifications(named: .GCSpatialAccessoryDidConnect) {
    if let accessory = notification.object as? GCSpatialAccessory {
        // Keep a reference to the accessory.
    }
}

Listen for disconnect notifications to release references when accessories disconnect:

for await notification in NotificationCenter.default.notifications(named: .GCSpatialAccessoryDidDisconnect) {
    if let accessory = notification.object as? GCSpatialAccessory {
        // Release the reference to the accessory.
    }
}

Load a 3D model of the accessory

The accessory manufacturer may bundle a USDZ of the accessory your app can render.

Create an AnchoringComponent.AccessoryAnchoringSource from the connected accessory to access the USDZ file:

let anchoringSource = try await AnchoringComponent.AccessoryAnchoringSource(device: accessory)

Use the source to load the USDZ as an Entity, falling back to a placeholder if it’s unavailable or loading fails:

let referenceEntity: Entity
if let usdzURL = anchoringSource.underlyingAccessory?.usdzFile {
    do {
        referenceEntity = try await Entity(contentsOf: usdzURL)
    } catch {
        logger.warning("Failed to load USDZ: \(error)")
        referenceEntity = makeFallbackEntity()
    }
} else {
    referenceEntity = makeFallbackEntity()
}

Anchor entities to named accessory locations

Use an AnchorEntity to render content that tracks an accessory. Create the anchor entity from the accessory’s anchoring source and a location on the device:

let anchorEntity = AnchorEntity(
    .accessory(from: anchoringSource, location: .origin),
    trackingMode: .continuous
)

You can anchor content to the origin of any generic accessory. Some accessories also let you anchor content to distinct points the manufacturer defines on the device. Query the supported named locations using the accessoryLocations property on AnchoringComponent.AccessoryAnchoringSource:

let availableLocations = anchoringSource.accessoryLocations

The tracking mode controls the trade-off between latency and accuracy. Use continuous for higher accuracy with increased latency, or predicted for lower latency with less accuracy.

Reading an anchor entity’s transform requires a running SpatialTrackingSession configured to track accessories. Entities parented to an AnchorEntity render at the accessory’s latest pose. Reading the transform programmatically yields a value one frame behind. Use AccessoryTrackingProvider, covered in the next section, for the lowest-latency transform access.

Track an accessory using ARKit

Use AccessoryTrackingProvider when your app needs the accessory’s transform in code. For example, when your app:

displays content in a volume, whose fixed bounds can prevent an anchored entity from following the accessory past the edges.
renders outside of RealityKit.
predicts the accessory’s position at a future timestamp.

Create an AccessoryTrackingProvider from one or more Accessory instances, then run the provider using an ARKitSession:

let trackedAccessory = try await Accessory(device: accessory)
let provider = AccessoryTrackingProvider(accessories: [trackedAccessory])
try await arkitSession.run([provider])

To respond to accessory anchor updates, iterate the provider’s anchorUpdates async sequence:

for await update in provider.anchorUpdates {
    switch update.event {
    case .added, .updated:
        let anchor = update.anchor
        // Use the anchor.
    case .removed:
        // Clean up the anchor.
    }
}

For the lowest-latency rendering, predict where the accessory will be at a future timestamp using predictAnchor(for:at:). Pass a timestamp offset that matches your rendering latency.

let predictedAnchor = provider.predictAnchor(
    for: anchor,
    at: CACurrentMediaTime() + renderLatencyCompensation
)

Read the world-space transform from the anchor using its coordinate space. Pass .rendered to align the transform with the rendering pipeline, or .none for the raw measured value:

let anchorSpace = predictedAnchor.coordinateSpace(correction: .rendered)
let worldTransform = Transform(projectiveTransform: anchorSpace.ancestorFromSpaceTransformFloat())

When the connected accessory changes, call updateAccessories(_:) on the running provider instead of stopping and restarting it:

let trackedAccessory = try await Accessory(device: accessory)
try await provider.updateAccessories([trackedAccessory])

For more information on using AccessoryTrackingProvider, see Drawing in the air and on surfaces with a spatial stylus.

For information on tracking in a volume, see Tracking accessories in volumetric windows. For information on using an accessory’s transform to drive interactive content, see Tracking a handheld accessory as a virtual sculpting tool.

Respond to accessory input

Some accessories have buttons. The accessory’s input property provides the familiar Game Controller interface.

For example, set an elementValueDidChangeHandler to respond to button presses:

guard let accessory = GCSpatialAccessory.spatialAccessories.first else { return }
accessory.input?.elementValueDidChangeHandler = { _, element in
    if let button = element as? GCButtonElement, button.pressedInput.isPressed {
        // Handle the button press.
    }
}

For more information on callback and polling approaches to input handling, see Handling input events.

Play haptic feedback

Some accessories support haptic feedback. The accessory’s haptics property provides access to Core Haptics.

For example, create a CHHapticEngine on the default locality:

guard let accessory = GCSpatialAccessory.spatialAccessories.first else { return }
let engine = accessory.haptics?.createEngine(withLocality: .default)
try await engine?.start()

With the engine running, play haptic patterns using Core Haptics.

For more information on input and haptics with accessories, see Discovering and tracking spatial game controllers and styli.