Working with generic spatial accessories

Let people place digital replicas of a generic spatial accessory by tracking the accessory with ARKit.

Overview

Specialized apps become more immersive when they respond to people using purpose-built devices, like medical instruments, steering rigs, or industrial tooling. ARKit recognizes these devices as generic spatial accessories, but they’re distinct from spatial controllers and styli, which have their own dedicated APIs.

Manufacturers create generic spatial accessories by following the Accessory Design Guidelines, paying close attention to the Spatial Accessories section. ARKit provides precise, low-latency tracking of these accessories in Apple Vision Pro across varied lighting conditions, and continues tracking orientation even when an accessory moves outside the field of view or becomes visually obscured. Game Controller also provides input and haptic feedback for these accessories.

In an immersive space, the sample lets people place digital replicas of the accessory. In a volume, the sample displays a digital replica that mirrors the accessory’s orientation in real time. Accessory manufacturers can also use the sample to test the tracking, input, and haptic behavior of their accessories.

Configure the sample code project

The sample works with any generic spatial accessory. Before building the sample, configure the Xcode project to access the .referenceaccessory file for the accessory you want to track. For step-by-step instructions, see Setting up access to a reference accessory file.

Add the accessory tracking capability

To help protect people’s privacy, visionOS limits app access to spatial accessory data and other sensor data on Apple Vision Pro. Add the Accessory Tracking capability to your app’s target and provide a usage description that explains how your app uses spatial accessory data. People see that description when the system prompts for access to accessory-tracking data. For more information on app capabilities, see Adding capabilities to your app.

Obtain authorization to track accessories

To read an accessory’s transform, your app needs Accessory Tracking authorization. The sample monitors the authorization status so it can change the availability of features that require the transform.

At startup, the sample uses an ARKitSession to query the current authorization status:

// AccessoryModel.swift

authorizationStatus = await arkitSession.queryAuthorization(for: [.accessoryTracking])[.accessoryTracking] ?? .notDetermined

The sample then observes authorization changes on the same session so the app can respond when people grant or revoke permission:

// AccessoryModel.swift

for await event in arkitSession.events {
    switch event {
    case .authorizationChanged(.accessoryTracking, let status):
        authorizationStatus = status
    default:
        break
    }
}

Discover connected accessories

Before your app can anchor content to an accessory, track it, or respond to its input, your app needs a reference to the GCSpatialAccessory instance for the connected accessory.

The sample uses a custom AccessoryModel to manage this reference, tracking the most recently connected accessory throughout the app’s life cycle.

At startup, AccessoryModel checks spatialAccessories for an already-connected accessory:

// AccessoryModel.swift

if let accessory = GCSpatialAccessory.spatialAccessories.first {
    self.accessoryDevice = accessory
}

To handle accessories that connect later, AccessoryModel listens for connect notifications:

// AccessoryModel.swift

private func observeAccessoryConnectNotifications() async {
    for await notification in NotificationCenter.default.notifications(named: .GCSpatialAccessoryDidConnect) {
        if let accessory = notification.object as? GCSpatialAccessory {
            self.accessoryDevice = accessory
        }
    }
}

To release the reference when the active accessory disconnects, AccessoryModel listens for disconnect notifications:

// AccessoryModel.swift

private func observeAccessoryDisconnectNotifications() async {
    for await notification in NotificationCenter.default.notifications(named: .GCSpatialAccessoryDidDisconnect) {
        if let accessory = notification.object as? GCSpatialAccessory,
           accessory == self.accessoryDevice {
            self.accessoryDevice = nil
        }
    }
}

Load a 3D model of the accessory

When a person opens the immersive space, the app displays a semi-transparent digital replica of the accessory. The accessory manufacturer can include a .usdz model of the accessory in the .referenceaccessory file. When an accessory connects, AccessoryModel retrieves the model through an AnchoringComponent.AccessoryAnchoringSource and loads it as an Entity. If the .referenceaccessory file doesn’t include a .usdz, or loading fails, AccessoryModel falls back to a placeholder entity:

// AccessoryModel.swift

private func loadReferenceEntity(from anchoringSource: AnchoringComponent.AccessoryAnchoringSource) async {
    if let usdzURL = anchoringSource.underlyingAccessory?.usdzFile {
        do {
            referenceEntity = try await Entity(contentsOf: usdzURL)
        } catch {
            logger.warning("Failed to load USDZ file, using missing reference entity fallback: \(error)")
            referenceEntity = Entity.createMissingReferenceEntity()
        }
    } else {
        referenceEntity = Entity.createMissingReferenceEntity()
    }
}

Anchor entities to named accessory locations

The app uses an AnchorEntity to keep the replica locked to the physical accessory’s origin as a person moves the accessory:

// ImmersiveView.swift

private func createReferenceAnchorEntity(
    for source: AnchoringComponent.AccessoryAnchoringSource,
    trackingMode: AnchoringComponent.TrackingMode
) -> Entity? {
    guard let referenceEntity = appModel.accessoryModel.referenceEntity else {
        return nil
    }

    let anchorEntity = AnchorEntity(
        .accessory(from: source, location: .origin),
        trackingMode: trackingMode,
        physicsSimulation: .none
    )
    anchorEntity.addChild(referenceEntity.clone(recursive: true))
    anchorEntity.components.set(OpacityComponent(opacity: 0.5))

    return anchorEntity
}

All generic accessories support anchoring entities to their origin, and some support anchoring to other named locations the manufacturer defines. The sample queries the accessoryLocations property on AccessoryAnchoringSource to discover an accessory’s supported locations, then presents a toggle for each in AccessorySettingsForm. When a person turns on a location toggle, the sample anchors a white sphere at that location.

The form also presents a picker for the tracking mode. The tracking mode controls the trade-off between latency and accuracy. Use continuous for higher accuracy with increased latency, or predicted for lower latency with less accuracy.

Read the accessory’s transform using RealityKit

When a person taps the “Place digital replica” button in ContentView, ImmersiveView reads the reference anchor entity’s transform, clones the reference entity at that position, and logs the underlying AccessoryAnchor to demonstrate how to access it:

// ImmersiveView.swift

private func placeDigitalReplica() {
    guard let referenceAnchorEntity = referenceRoot.children.first else { return }

    guard let referenceEntity = appModel.accessoryModel.referenceEntity else {
        // ...
        return
    }

    let replica = referenceEntity.clone(recursive: true)
    replica.transform.matrix = referenceAnchorEntity.transformMatrix(relativeTo: nil)

    if let arkitComponent = referenceAnchorEntity.components[ARKitAnchorComponent.self],
       let underlyingAnchor = arkitComponent.anchor as? AccessoryAnchor {
        // Access the anchor associated with an anchor entity.
        logger.debug("Underlying anchor: \(underlyingAnchor)")
    }

    digitalReplicaRoot.addChild(replica)
}

Reading an anchor entity’s transform requires a running SpatialTrackingSession configured to track accessories.

Entities parented to an AnchorEntity always render at the accessory’s latest pose. When placing a digital replica, the sample instead clones the reference entity and applies the anchor entity’s transform to the clone. This pattern introduces a one-frame lag, because by the time RealityKit renders the clone, the accessory has moved to a new pose. This approach is simple and works well when the one-frame lag is acceptable. For content that needs continuous low-latency tracking, see the Track an accessory using ARKit section.

Track an accessory using ARKit

Use AccessoryTrackingProvider when your app needs to choose how and when it reads the accessory’s transform, or when it needs more control over rendering than RealityKit provides. For example, when your app:

displays content in a volume, whose fixed bounds can prevent an anchored entity from following the accessory past the edges.
renders outside of RealityKit.
predicts the accessory’s position at a future timestamp.

The AccessoryModel creates an AccessoryTrackingProvider for the connected Accessory and runs it on an ARKitSession:

// AccessoryModel.swift

let accessory = try await Accessory(device: accessoryDevice)
let provider = AccessoryTrackingProvider(accessories: [accessory])
try await arkitSession.run([provider])

The connected accessory may change during the app’s life cycle. When it does, the sample updates the running provider rather than stopping and restarting it, using updateAccessories(_:):

// AccessoryModel.swift

let accessories = await buildAccessoriesForTrackingProvider()
try await accessoryTrackingProvider.updateAccessories(accessories)

With the provider running, the app can respond when a person moves the accessory. When a person taps the “Show the Volume” button in ContentView, the app displays VolumeView containing a digital replica that rotates in real time to mirror the physical accessory’s orientation:

// VolumeView.swift

RealityView { content in
    guard let sourceEntity = appModel.accessoryModel.referenceEntity else { return }
    let clone = sourceEntity.clone(recursive: true)

    content.add(clone)
    // ...
    scaleToFit(clone)

    subscription = content.subscribe(to: SceneEvents.Update.self) { _ in
        if let accessoryAnchor = appModel.accessoryModel.queryLatestAccessoryAnchor(),
           let rotation = accessoryAnchor.coordinateSpace(correction: .rendered).ancestorFromSpaceTransformFloat().rotation {
            clone.transform.rotation = unsafe rotation.quaternion
        } else {
            clone.transform.rotation = simd_quatf()
        }
    }
}

Typically, apps iterate the provider’s anchorUpdates to react to accessory anchor updates, for example:

// Reference snippet: This isn't part of the project's sample code.

for await update in provider.anchorUpdates {
    switch update.event {
    case .added, .updated:
        let anchor = update.anchor
        // Use the anchor.
    case .removed:
        // Clean up the anchor.
    }
}

However, the sample takes a different approach. Rather than relying on anchorUpdates, this project subscribes to SceneEvents.Update from a render loop in RealityKit. This bypasses an update cycle in SwiftUI which otherwise invalidates views at the provider’s high frame rate.

To keep the replica’s rotation in sync with the accessory, VolumeView reads the accessory’s pose each frame by calling queryLatestAccessoryAnchor. In predicted mode, queryLatestAccessoryAnchor uses predictAnchor(for:at:) to estimate the accessory’s pose a few frames into the future:

// AccessoryModel.swift

func queryLatestAccessoryAnchor() -> AccessoryAnchor? {
    guard let accessoryTrackingProvider,
          accessoryTrackingProvider.state == .running,
          let latestAnchor = accessoryTrackingProvider.latestAnchors.first else {
        return nil
    }

    if trackingMode == .predicted {
        return accessoryTrackingProvider.predictAnchor(for: latestAnchor, at: CACurrentMediaTime() + renderLatencyCompensation)
    }

    return latestAnchor
}

The VolumeView passes ARKitCoordinateSpace.Correction.rendered to coordinateSpace because it applies the rotation to a rendered entity. A non-rendering use case, like a measuring app, would pass ARKitCoordinateSpace.Correction.none to read the raw rotation.

The VolumeView only reads the rotation from the accessory’s transform. To obtain the accessory’s full transform, wrap ancestorFromSpaceTransformFloat() in a Transform:

// Reference snippet: This isn't part of the project's sample code.

let worldTransform = Transform(
    projectiveTransform: accessoryAnchor
        .coordinateSpace(correction: .rendered)
        .ancestorFromSpaceTransformFloat()
)

For more information on using AccessoryTrackingProvider, see Drawing in the air and on surfaces with a spatial stylus.

For information on tracking in a volume, see Tracking accessories in volumetric windows. For information on using an accessory’s transform to drive interactive content, see Tracking a handheld accessory as a virtual sculpting tool.

Respond to accessory input

Some accessories have buttons. The accessory’s input property provides the familiar Game Controller interface. If the connected accessory has buttons, a person can press any button to initiate the same functionality as the “Place digital replica” button in ContentView.

When an accessory connects, AccessoryModel sets an elementValueDidChangeHandler on the accessory’s input to call initiateDigitalReplicaPlacement on every button press:

// AccessoryModel.swift

private func handleAccessoryDeviceChange() {
    // ...

    accessoryDevice.input?.elementValueDidChangeHandler = { [weak self] (_, element) in
        guard let self else { return }
        if let button = element as? GCButtonElement,
           button.pressedInput.isPressed {
            logger.info("Button pressed: \(element.localizedName ?? "Unnamed element")")
            initiateDigitalReplicaPlacement()
        }
    }

    // ...
}

For more information on callback and polling approaches to input handling, see Handling input events.

Play haptic feedback

Some accessories support haptic feedback. The accessory’s haptics property provides access to Core Haptics. If the connected accessory supports haptics, a person can tap the “Play haptics” button in ContentView to provide feedback on the accessory.

When HapticModel initializes, it creates a CHHapticEngine on the accessory’s default locality and starts it:

// HapticModel.swift

init(accessory: GCSpatialAccessory) async {
    guard let engine = accessory.haptics?.createEngine(withLocality: .default) else {
        logger.info("The accessory doesn't support haptics.")
        return
    }

    do {
        try await engine.start()
        hapticEngine = engine
        logger.info("The haptic engine started successfully.")
    } catch {
        logger.error("Failed to start the haptic engine: \(error)")
        hapticEngine = nil
    }
}

With the engine running, the sample plays haptic patterns using CHHapticPattern whenever a person taps “Play haptics”.

For more information on input and haptics with accessories, see Discovering and tracking spatial game controllers and styli.