Analyzing the performance of a foveated streaming session

Use the Foveated Streaming Statistics instrument to evaluate the performance of your visionOS streaming client app.

Overview

Maintaining a high frame rate while minimizing latency is critical to ensure a comfortable and immersive foveated streaming experience on Apple Vision Pro. The Foveated Streaming framework delivers high-quality immersive content to Apple Vision Pro through a highly-optimized, real-time pipeline:

Apple Vision Pro sends its pose and information about the approximate region where the person is looking to a streaming server.
The server uses the pose to render a frame.
The server foveates that frame for streaming, applying encoding at higher resolution only where necessary.
The server sends the foveated frame back to Apple Vision Pro for decoding and display.

This intricate, real-time process demands minimal latency to mitigate motion sickness, disorientation, and breaks in immersion that high latency causes.

To get useful information specific to foveated streaming, profile your app with the Foveated Streaming Statistics instrument. This instrument helps you identify:

Available network bandwidth and streaming rate.
Received and reprojected frame rates.
End-to-end streaming latency.

For more information on how to use Instruments, see Profiling apps using Instruments.

Prepare the foveated streaming statistics instrument

To set up a new Instruments document from Xcode:

Choose Product > Profile; this opens the Instruments app.
In Instruments, select the Blank template, then click Choose.
In the Instruments selector, click “+ Instrument”.
In the filter field, enter “fov” to quickly locate the Foveated Streaming Statistics instrument.
Double click the instrument to add the Foveated Streaming Statistics instrument.

[Image]

Capture streaming statistics

To collect metrics for a foveated streaming session:

Click the Record button at the top left of the window to start capturing profile data.
Connect the FoveatedStreamingSession in your app to a streaming endpoint that’s actively streaming content.
Click the stop button to stop recording.

[Image]

The Foveated Streaming Statistics instrument collects metrics on the following:

Average Streaming Rate: The amount of data being streamed per second.
Available Bandwidth: The total network bandwidth available for streaming. This acts as the upper limit on the Average Streaming Rate.
Frame Receive Rate: The number of frames received from the streaming server per second.
Reprojection Rate: The frame rate the app displays the received frames with. For example, if the Frame Receive Rate is lower than the maximum frame rate of the display, Apple Vision Pro re-projects the received frames to improve the overall frame rate.
Pose Rate: The sampling rate of the device pose that Apple Vision Pro sends to the streaming server.
Pose to Frame Submitted Latency: The end-to-end latency from Apple Vision Pro sending the device pose to receiving, decoding, and submitting a streamed frame to display.
Pose to Frame Dequeued Latency: The latency from Apple Vision Pro sending the device pose to receiving an encoded frame and dequeing it from a buffer to compensate for network timing fluctuations.
Pose to Frame Received Latency: The latency from Apple Vision Pro sending the device pose to receiving an encoded frame.
Pose Upload Latency: The one-way latency between Apple Vision Pro sending the device pose and the server receiving it.

Diagnose performance issues

The most common source of foveated streaming performance issues is network limitations. If you observe visual artifacts, stuttering, or a lack of responsiveness, begin by examining the Available Bandwidth and Average Streaming Rate. A significant gap between these two values, or consistently low numbers for either, strongly indicates that your network connection is the limiting factor. This could be due to network congestion, poor Wi-Fi signal strength, or other connectivity problems.

Evaluate your app’s performance

If the Available Bandwidth and Average Streaming Rate appear reasonable, the next step is to investigate your app’s internal performance. One indicator of your app’s performance is the Pose Rate. For optimal foveated streaming, the Pose Rate should be consistent with the device’s refresh rate; for example, 90 Hz. A Pose Rate significantly below this target suggests that your app isn’t running at its maximum frame rate. This indicates that something within your app’s logic or rendering pipeline is causing slowdowns.

In such cases, you can use other Xcode instruments, like Time Profiler, to pinpoint the specific performance bottlenecks within your app.

To understand the overall responsiveness and end-to-end latency of the system, monitor the Pose to Frame Submitted Latency metric. This metric measures the total time from when Apple Vision Pro sends a device pose to when decoding and submitting the corresponding streamed frame for display.

You want this latency to be as low as possible for a fluid and reactive experience. Consistently high Pose to Frame Submitted Latency — even when initial network and streaming rates seem adequate — can suggest more subtle or intermittent network issues, such as network jitter or packet loss, which might not be immediately obvious from average bandwidth metrics.

If this latency is high, focus on optimizing your network environment for stability and consistency, or review your app’s processing and rendering pipeline for any client-side work that could introduce delays before frame submission.