--- title: Implementing object tracking in your app framework: visionos role: article role_heading: Article path: visionos/implementing-object-tracking-in-your-app --- # Implementing object tracking in your app Create engaging interactions by training models to recognize and track real-world objects in people’s surroundings. ## Overview Overview When implementing object tracking in your iOS or visionOS app, you can seamlessly integrate real-world objects in people’s surroundings to enhance their immersive experiences. By tracking the 3D position and orientation of an object, or several objects, your app can augment the objects with virtual content and measure the relative spatial distance between them. Use object tracking to provide virtual interactions with objects in a person’s surroundings, such as: Verifying that a tool is held and positioned correctly during an assembly process. Guiding someone through using an item’s features, reading about its history, or learning about its behaviors when they look at it in their surroundings. Helping people troubleshoot issues with household items and appliances with a virtual manual. Creating an immersive storytelling experience to make collectables and toys come to life. To add object tracking to your app, start with a physical object’s 3D model, train a machine learning model in Create ML with that 3D model to obtain a reference object file, then use the resulting reference object file to track the physical object. The reference object file has a .referenceobject file extension, specifically for object tracking in iOS or visionOS. Implementing object tracking requires either an iPhone with iOS 27 or later, or an Apple Vision Pro with visionOS 2 or later. The machine learning training in Create ML requires a Mac with Apple silicon and macOS 15 or later. Reference object files trained with macOS 27 and Xcode 27 or later require iOS 27 or visionOS 27 and later. Ensure your objects are suitable for object tracking For object tracking to work best in your app, make sure your object is rigid. A rigid object maintains its shape and appearance during tracking. For example, a pair of scissors is challenging to track because it changes shape while a person uses it. By default, object tracking delivers pose updates at a lower frequency and consumes less power. This works well for stationary objects, or cases where virtual content doesn’t need to match the object’s exact position on every frame. For moving or handheld objects, like surgical instruments or power tools, set highFrameRateTrackingEnabled when you create the object tracking session in visionOS, or assign the object to trackingObjects in iOS. This gives you a precise pose update on every frame. For best results, train your reference object in extended mode. For more information, see Train a machine learning model with the 3D model asset in Create ML later in this article. Obtain a 3D model of your object You use Create ML to begin the machine learning training to obtain your reference object file. Create ML requires a 3D model asset in the USDZ file format that represents your real-world object. You can obtain your 3D model using computer-aided design (CAD) software to accurately model an object’s geometry and apply physically based rendering (PBR) materials to it, and save it in the USDZ file format. Using this method, the 3D model can realistically represent objects that consist of multiple parts made from different materials, like glass, metal, plastic, wood, and other common materials. This method is helpful for capturing objects that are entirely or partly transparent, shiny, or reflective. The better the 3D model represents the appearance of the physical object, the better the quality of tracking is in visionOS. Another way to create your 3D model is by using the Object Capture feature in the Reality Composer app in iOS or iPadOS. You can use your iPhone or iPad to capture images of an object, and then save the USDZ file to import into your app. For more information about using the Object Capture feature to create a 3D model, see Meet Object Capture for iOS and Scanning objects using Object Capture. Before beginning the training process in Create ML with the 3D model asset, keep the following guidelines in mind to ensure it works well for object tracking in visionOS: Ensure the 3D model is as photorealistic as possible — essentially a digital twin of your real-world object. Ensure the scale of the 3D model is as precise as possible and matches its specified units. If the scale doesn’t match the real-world object, the augmentation appears offset in the viewing direction, and may appear either in front of or behind the object. note: While training the machine learning model with the 3D model asset, Create ML ignores any animations, virtual cameras, and lights within the asset, treating them as static. Train a machine learning model with the 3D model asset in Create ML Object tracking requires a reference object file to track the spatial location and orientation of the corresponding real-world object. You use Create ML to train a machine learning model to create a reference object file unique to your object. The training of machine learning models with your 3D asset and the creation of the reference object file both run locally on your Mac. You can either train a model with the Create ML developer tool that comes with Xcode, or with the Create ML command-line tool. The following are the steps to train a model in the Create ML app: Open Xcode and choose Xcode > Open Developer Tool > Create ML. Create a new project or open an existing project. In the Choose a Template dialog, select Object Tracking template, which is in the Spatial category, and click Next. Give your project a name. Optionally, add a description or license to the model. Click Next. Choose where to save your project and click Create. Create ML opens a training configuration view with an empty 3D viewport. Drag the USDZ file of your 3D model asset into the 3D viewport. The 3D viewport is an interactive space where you can view your 3D model asset from different angles. After it appears in the viewport, check the appearance of the 3D model asset and confirm that it matches the absolute dimensions of your real-world object. Also make sure that the dimensions of the 3D model asset at the bottom right of the viewport match the actual dimensions of your object. If the scale doesn’t match, one option is to use Reality Composer Pro to rescale the 3D model and then add the adjusted USDZ file to Create ML. The next step is to select the best viewing angle for your real-world object. Consider how people view and interact with the object in your app, and decide which angle you need for tracking it. The “Viewing angles” setting appears below the 3D viewport, and has three viewing angles you can use: All Angles, Upright, or Front. note: Only choose the All Angles option if you want to track your object from all sides. Tracking is more accurate for objects trained with restricted viewing angles. After selecting a viewing angle, choose a training mode. Create ML offers two training modes, standard and extended: note: If you plan to track the object using high frame rate tracking, training with extended mode is recommended. If there’s an object in a person’s surroundings that’s similar to the object you want to track, the object-tracking feature might recognize and track it instead of your object. To prevent this from happening, add the similar object as a negative example when training the machine learning model with your reference object. Below the 3D viewport, choose More Options > Objects to avoid. Use this section to add USDZ samples of similar items to ensure the machine learning model doesn’t identify them as the object you want to track. Create ML supports training multiple machine learning models in the same object-tracking project. In the Model Sources section in the left pane, you can click the Add button (+) to add more 3D model assets to your Create ML project. Use this feature to track multiple objects in your app at the same time. After inspecting your 3D model asset and configuring the training settings, click Train to begin the training process. A progress bar indicates the amount of time until the machine learning training is complete. The machine learning training can take a few hours, depending on the configuration of your Mac. A more advanced processor and additional RAM significantly improve the training time. Train your assets with the Create ML command-line tool Starting with Xcode 26, which requires macOS 15.4 or later, you can train a machine learning model with your 3D asset by running the Create ML developer tool from a command line prompt. With an asset in the USDZ file format, you can use the tool to train the asset and get a reference object file to use for object tracking. The Create ML command-line tool automates object tracking tasks in your workflow, like using your scripts and cloud-based parallel setups to run the training process. You can also use the tool when you need to automate training a large number of objects while you continue to work on other tasks. You need to have Xcode command-line tools installed before using the Create ML tool, which you can check by running the following command: % xcode-select -p note: You can use the Create ML command line tool if the command’s output refers to a Contents/Developer directory in Xcode or to the /Library/Developer/CommandLineTools directory. If the command returns an error that indicates that Xcode or the command-line tools can’t find the directory, install the command-line tools package by running the xcode-select --install command. Begin the training process by invoking the Create ML command-line tool with the xcrun command. You need to modify the example below to provide the locations on your system for the commands inputs and outputs. % xcrun createml objecttracker -s source.usdz -o tracker.referenceobject The system uses the xcrun prefix to locate the path of the training tool in the Xcode command-line tools. The -s option points to the source path for the 3D asset of the physical object you want to train, and the -o option points to the output path to store the final trained reference object file. Before running this command, update it to include the name of the source and output of your object. By default, the tool trains the asset in standard mode. To use extended training mode, pass the -m option with the mode specified as extended: % xcrun createml objecttracker -s source.usdz -o tracker.referenceobject -m extended --all-angles For more information on training modes, see Train a machine learning model with the 3D model asset in Create ML earlier in this article. After you run the tool, it starts training your object. Use the help option (-h) for more information on training and topics like viewing angles, objects to avoid, and redirection to alternative pipes: % xcrun createml objecttracker -h Export the reference object file When training is complete, Create ML provides the reference object file for you to use in your app. Click the Output tab and save the resulting reference object file. The reference object file contains the machine learning model you trained, packaged with the 3D model asset, in the USDZ file format. You can use the USDZ file for visualizing the tracking quality by rendering it as an overlay on the real-world object, and as a guide for adding immersive effects. The USDZ file may take up a lot of space in your app if your 3D model asset is large, so you can remove it from the reference object file if you need to optimize space. Use the Reference Object Compiler in Xcode to remove the USDZ data from the reference object file during the build process. Select your project in Xcode, click the Build Settings tab, and enable the Strip USDZ Files from Reference Object option. This setting contains the REFERENCEOBJECT_STRIP_USDZ build option. The default setting of the option is No, so Xcode copies any reference object files you add to the project as-is unless you change the setting. Integrate the reference object file into your app After generating the reference object file, set up object tracking in your app using Reality Composer Pro, RealityKit, or ARKit. For more information about each of these methods, see Using a reference object with Reality Composer Pro, Using a reference object with RealityKit, Using a reference object with ARKit in visionOS, and Using a reference object with ARKit in iOS. important: In a visionOS app, object tracking works only in an ImmersiveSpace. Attempting to use object tracking in a window or volume results in a silent failure. For more information about object tracking, watch the WWDC24 session, Explore object tracking for visionOS. For an example of using ARKit for object tracking, see Exploring object tracking with ARKit. ## Topics ### Object tracking within an app - [Using a reference object with Reality Composer Pro](visionos/using-a-reference-object-with-reality-composer-pro.md) - [Using a reference object with RealityKit](visionos/using-a-reference-object-with-realitykit.md) - [Using a reference object with ARKit in visionOS](visionos/using-a-reference-object-with-arkit-in-visionos.md) - [Using a reference object with ARKit in iOS](visionos/using-a-reference-object-with-arkit-in-ios.md) ## See Also ### RealityKit and Reality Composer Pro - [Reality Composer Pro](realitycomposerpro.md) - [Petite Asteroids: Building a volumetric visionOS game](visionos/petite-asteroids-building-a-volumetric-visionos-game.md) - [BOT-anist](visionos/bot-anist.md) - [Swift Splash](visionos/swift-splash.md) - [Diorama](visionos/diorama.md) - [Building an immersive media viewing experience](visionos/building-an-immersive-media-viewing-experience.md) - [Enabling video reflections in an immersive environment](visionos/enabling-video-reflections-in-an-immersive-environment.md) - [Combining 2D and 3D views in an immersive app](realitykit/combining-2d-and-3d-views-in-an-immersive-app.md) - [Understanding the modular architecture of RealityKit](visionos/understanding-the-realitykit-modular-architecture.md) - [Using transforms to move, scale, and rotate entities](visionos/understanding-transforms.md) - [Capturing screenshots and video from Apple Vision Pro for 2D viewing](visionos/capturing-screenshots-and-video-from-your-apple-vision-pro-for-2d-viewing.md) - [Placing entities using head and device transform](visionos/placing-entities-using-head-and-device-transform.md) - [Manipulating entities with solid collisions](visionos/manipulating-entities-with-solid-collisions.md) - [Gaussian splats on visionOS](visionos/gaussian-splats-on-visionos.md)