Contents

Streaming a CloudXR application to Apple Vision Pro with foveation

Integrate NVIDIA CloudXR™ and the session management connection protocol into your desktop or cloud application to stream high-fidelity spatial content to Apple Vision Pro.

Overview

If you have an existing virtual reality game, experience, or application built for desktop computers or a cloud server, you can stream it to Apple Vision Pro with the Foveated Streaming framework.

Foveated Streaming allows your endpoint to stream high quality content only where necessary based on information about the approximate region where the person is looking, ensuring performance. On Apple Vision Pro, you can also layer native spatial content over the streamed content. For example, a racing game can render the gauges in the interior of the car with RealityKit, and stream the processor-intensive outdoor environment from a remote computer to the device.

To enable streaming, integrate the following three components:

  • NVIDIA CloudXR Runtime: Download the OpenXR-compliant runtime from NVIDIA and integrate it into your streaming application.

  • Session management connection protocol: When streaming from a local device, implement a protocol to handle device discovery, authentication, and session state management between your streaming application and Apple Vision Pro.

  • Apple Vision Pro client app: Build a visionOS app that connects to your streaming application, displays the streamed content in an immersive space, and enables people to interact with both the streamed content and RealityKit elements.

Add the CloudXR runtime to your application

Download the CloudXR SDK, then follow the steps in the NVIDIA CloudXR SDK documentation to integrate it into your app.

For an example implementation, see the StreamingSession-OpenXRSample.

Implement the session management connection protocol

Make your streaming application discoverable to Apple Vision Pro by broadcasting a Bonjour service with the following name:

_apple-foveated-streaming._tcp

Add a TXT record to the Bonjour service with an entry that specifies the bundle ID of the visionOS app you’re streaming to:

Key

Value

Application-Identifier

The bundle ID of the visionOS app.

After discovering the streaming endpoint, Apple Vision Pro sends and expects replies to a series of Transmission Control Protocol (TCP) messages. The order of these messages is as follows:

  1. Apple Vision Pro sends a RequestConnection message.

    • Your streaming application replies with an AcknowledgeConnection message.

  2. Apple Vision Pro sends a RequestBarcodePresentation message.

    • Your streaming application presents a QR code with pairing information and replies with AcknowledgeBarcodePresentation.

  3. After launching the CloudXR server, your streaming application sends a MediaStreamIsReady to the server to indicate that it’s ready to stream content.

  4. Apple Vision Pro sends SessionStatusDidChange messages when the foveated streaming session state changes.

  5. Your streaming application sends a RequestSessionDisconnect message to end the session.

For more information on the format and content of these messages, see Establishing foveated streaming sessions with Apple Vision Pro. For an example implementation, see the StreamingSession-WindowsApp.

Create a visionOS app

Display streamed content by building a visionOS app with the Foveated Streaming framework that does the following:

You can use the Foveated Streaming App template in Xcode to get started quickly. To do so, create a new project in Xcode by choosing File > New > Project. Navigate to the Multiplatform section of the template chooser, and select the Foveated Streaming App template:

[Image]

When prompted, specify a name for your project along with other options.

Alternatively, start from the Foveated Streaming sample app. For more information, see Creating a foveated streaming client on visionOS.

See Also

Essentials