Animating hand models in visionOS

Overview

Hand tracking enables your app to observe an Apple Vision Pro wearer’s hand positions and joint rotations in real-time within an immersive space. This sample code project demonstrates how to replace the wearer’s hands with animated glove models in a fully immersive space.

Create the glove models

The glove models in this sample code project match the hand skeleton structure in ARKit. Each glove’s origin is positioned at the base of the wrist just like the ARKit hand skeleton’s origin. Each glove has 27 joints, matching both the number and order of the ARKit hand skeleton. These joints form a kinematic chain where each joint’s position and rotation is relative to the preceding joint, extending from wrist to fingertips. For example, the wrist connects to the thumb base, which connects to the first thumb joint, and so on.

The gloves use a natural, relaxed hand pose as the rest pose. The fingers point forward along the negative z-axis, the palm faces downward along the negative y-axis, the thumb extends at approximately 45 degrees from the palm, and the fingers curl slightly rather than fully extend. This rest pose aligns with the ARKit hand skeleton, ensuring natural animation when applying joint rotations.

[Image]

For more information on ARKit in visionOS, including hands-tracking, see Meet ARKit for spatial computing.

Hide the wearer’s hands

By default, the system displays the wearer’s hands in an immersive space. The sample hides the wearer’s hands using doc://com.apple.documentation/documentation/swiftui/scene/upperlimbvisibility(_:).

ImmersiveSpace(id: appModel.immersiveSpaceID) {
    ImmersiveView()
        
    // ...    
}
.immersionStyle(selection: .constant(.full), in: .full)
.upperLimbVisibility(.hidden)

Add usage descriptions for ARKit data access

To help protect people’s privacy, visionOS limits app access to hands-tracking data and other sensors in Apple Vision Pro. Add the Hands Tracking capability to your app’s target and provide a usage description that explains how your app uses the data those sensors provide. People see that description when your app prompts for access to hands-tracking data. For more information on app capabilities, see Adding capabilities to your app.

Add the glove models and start hand tracking

The ImmersiveView loads the glove models, adds them to the RealityKit scene, and starts hand tracking.

@State private var arkitSession = ARKitSession()
@State private var handTrackingProvider = HandTrackingProvider()
@State private var leftGlove: ModelEntity?
@State private var rightGlove: ModelEntity?

var body: some View {
    RealityView { content in
        let root = Entity()
        
        if let glove = await loadGloveModel(named: "LeftGlove") {
            leftGlove = glove
            root.addChild(glove)
        }
        
        if let glove = await loadGloveModel(named: "RightGlove") {
            rightGlove = glove
            root.addChild(glove)
        }
        
        content.add(root)
    }
    .task {
        await startHandTracking()
    }
}

The startHandTracking method runs an ARKitSession with a HandTrackingProvider to begin hand tracking, then responds to updates from the HandTrackingProvider.

private func startHandTracking() async {
    do {
        try await arkitSession.run([handTrackingProvider])
    } catch {
        print("Failed to start hand tracking: \(error.localizedDescription).")
    }
    
    await updateGlovesFromHandAnchors()
}

Animate the glove models

ARKit delivers a stream of HandAnchor instances, each containing skeletal tracking data for one of the wearer’s hands. The updateGlovesFromHandAnchors method hides the glove when ARKit is unable to track the hand. Each hand anchor includes originFromAnchorTransform, a transform with an origin at the base of the wrist. Because the glove model’s origin is also at the wrist, updateGlovesFromHandAnchors sets the glove’s transform to match this transform, positioning the glove correctly in space.

private func updateGlovesFromHandAnchors() async {
    for await anchorUpdate in handTrackingProvider.anchorUpdates {
        let handAnchor = anchorUpdate.anchor
        
        guard let glove = handAnchor.chirality == .left ? leftGlove : rightGlove,
              let handSkeleton = handAnchor.handSkeleton else { continue }
        
        // Hide the glove when the system loses tracking.
        glove.isEnabled = handAnchor.isTracked
        
        guard handAnchor.isTracked else { continue }
        
        glove.transform = Transform(matrix: handAnchor.originFromAnchorTransform)
        updateJointRotations(for: glove, using: handSkeleton)
    }
}

Finally, the updateJointRotations method iterates through each joint, extracts its rotation from the hand skeleton, and applies that rotation to the corresponding joint in the glove model. This iteration by index requires that the joints in the USD model have the identical order as the ARKit hand skeleton. Because the glove model uses the same joint hierarchy as the ARKit hand skeleton, applying these rotations updates the glove’s pose to match the wearer’s hand pose.

private func updateJointRotations(for glove: ModelEntity, using handSkeleton: HandSkeleton) {
    let joints = handSkeleton.allJoints
    
    // This assumes the joint order in the USD file matches the ARKit hand skeleton joint order.
    for (index, joint) in joints.enumerated() {
        let jointTransform = handSkeleton.joint(joint.name).parentFromJointTransform
        let rotation = simd_quatf(jointTransform)
        
        glove.jointTransforms[index].rotation = rotation
    }
}

See Also

Integrating ARKit

• Creating a 3D painting space
• Tracking and visualizing hand movement
• Displaying an entity that follows a person’s view
• Applying mesh to real-world surroundings
• Obscuring virtual items in a scene behind real-world items