visionOS Group Lab

Transcript

Engineers from the visionOS team introduce themselves and set up a session covering Reality Composer Pro, RealityKit and USD, camera access, Gaussian splatting, spatial accessories, Visual Intelligence, debugging on device, and agentic coding for visionOS.

Apple opened camera access progressively — first the left main camera, then the stereoscopic pair in visionOS 26. Access is granted through an application process where you demonstrate what your app will do, oriented mainly toward professional and enterprise use cases. Importantly, this isn't limited to the Apple Developer Enterprise Program — developers with a standard Apple Developer Program account can apply too (typically where the account is attached to an incorporated business). If you have a real use case and can't get access today, post in the developer forums and file Feedback describing it.

There's a developer setting, added last year, that lets you use Mac Virtual Display inside an immersive environment. So if you're building an app that surrounds you in a virtual world, you can build and run from Xcode with Mac Virtual Display open alongside your immersive content — when you hit a breakpoint you stay where you are (which also matters for safety) and can triage and debug in place. Internally this is the common workflow: work in Vision Pro with Mac Virtual Display and immersive content side by side, rather than repeatedly donning and doffing the headset — and it doubles as an enormous, high-quality display.

There are structural differences between an iOS app and a visionOS app — form factor, size, information density. Of the skills shipped so far, the most directly relevant helps resize an existing iOS app to visionOS, translating the UI in a way that maintains visionOS best practices and principles. If you're bringing a 2D iOS app over, that's the skill to reach for.

Start in your 3D DCC tool and export to USD as directly as possible, since round-tripping through intermediate formats (FBX/GLB) is where transforms and materials get altered. Validate the result in Reality Composer Pro, and rebuild materials there when a complex model's shading doesn't survive the export.

Spatial accessory tracking isn't limited to the field of view — the accessory tracking APIs can follow accessories using their own sensors/signals rather than relying solely on the headset cameras seeing them, which makes out-of-view uses like foot trackers feasible. File Feedback with the specific use case to help shape the capability.

There's no known way to disable that near-plane culling/vignette today. The team found the telepresence reconstruction use case compelling and asked the developer to file Feedback so they can consider supporting it, since it's not currently configurable.

If the documentation indicates iOS support but the symbols aren't surfacing in Xcode, that's worth a Feedback report so the team can investigate the discrepancy. The panel encouraged filing it with specifics rather than assuming, since availability and symbol exposure across platforms is something they can clarify and fix.

Vision Pro is designed primarily as a personal device tuned to one wearer (eye and hand calibration, Persona, Optic ID), which is why full multi-user profiles aren't offered. The team acknowledged strong interest in device sharing and treated it as valuable feedback, while noting Guest User exists for occasional shared use today.

ARKit's face tracking is an iOS-only feature and isn't available on visionOS, and there's no API exposing raw face data on Vision Pro — so you can't drive a custom face-mesh avatar or apply lens-style effects to the wearer's face. What you can get as a third-party app is a matted version of the user's Persona — a video feed of their Persona over a background, similar to a front-facing camera feed — which works well for something like a video-conferencing app. File Feedback if you need more.

Visual Intelligence is handled at the system level — when Siri answers questions about what the user is looking at, it operates through the system rather than requiring your app to obtain main camera access. Third-party apps don't need camera entitlements to benefit from the system-provided Visual Intelligence experience.

Eyes, hands, and voice remain the primary interaction model; game controllers are a complementary input for scenarios that benefit from precise or sustained control. Support the standard Game Controller framework where it improves your app, but design so controllers augment rather than replace the natural spatial inputs, keeping the core experience usable without one.

The team pointed to WebKit/Safari's ongoing WebXR investment and encouraged filing Feedback for the specific immersive-ar limitation, since developer demand and concrete app examples help prioritize expanding the supported WebXR session modes on visionOS.

The panel declined to detail the internal implementation, but framed Visual Intelligence as a system capability you don't need to reverse-engineer — you get its results through the system rather than by accessing the underlying sensors yourself. Focus on the API surface Apple exposes rather than the sensor fusion behind it.

The auto-sleep on removal is intentional (privacy and safety), so there isn't a blessed "stay awake off-head" mode. During development, keep the device on your head or supported on your face for the sensors, and structure debugging so you don't need it idle on a desk — the team took the development-ergonomics request as feedback.

For RealityKit, the RealityRenderer class is a useful way to drive a lot of rendering through an automated test programmatically. And because RealityKit is written in Swift, your existing Swift and Xcode testing infrastructure works out of the box. Two sessions this year are worth watching: Migrate to Swift Testing, and Get the most out of Device Hub — you can do a surprising amount of automated testing through Device Hub.

A common one is porting iOS assumptions directly instead of designing for spatial computing — over-using flat windows, ignoring depth and comfort, and not considering how content sits in the user's space. Start from the platform's spatial design principles (comfortable placement, appropriate scale, respecting the user's surroundings) rather than treating visionOS as a big iPad screen.

LLM-generated code can contain errors, so be deliberate about how you prompt and always double-check the output — trust but verify. Because APIs and SDK capabilities evolve every year, an existing model may have been trained on previous years' APIs, so newer visionOS syntax like RealityView is a common gap. A practical fix is to give the model good context: point it at Apple's library of sample projects — Hello World, Petite Asteroids, Canyon Crosshair, and the Model Manipulator sample — so it has correct, current examples to learn from. The Collaborate on structured 3D models in visionOS session and its Model Manipulator sample are especially useful and exercise new RealityKit APIs like cross-sectioning.

Object tracking has improved dramatically. In visionOS 2.0 it focused on mostly-stationary objects at around 5 frames per second; new in visionOS 27, a high-frame-rate mode runs up to 30 frames per second at significantly lower latency, with recent smoothing work that makes a tracked object feel genuinely attached to your hand — even a fully passive object with no onboard electronics. Updates now carry frame-time-aligned timestamps, removing the plus-or-minus-one-frame ambiguity. And for the first time, in iOS 27 the same trained models also run on iOS devices, with a new Create ML extended training mode for higher accuracy and lower latency. See the developer documentation for specifics.

The team welcomed the accessibility telepresence use case and treated it as strong feedback, but couldn't commit to exposing those internal frameworks. The guidance is to file detailed Feedback describing the accessibility scenarios, since concrete developer needs inform what capabilities get opened up over time.

Mac Virtual Display's pairing is built around the personal-device model, so driving multiple shared Macs isn't directly supported today. The team took the enterprise/shared photogrammetry workflow as valuable feedback and encouraged filing it, since expanded shared-device support depends on demand like this.

It's partially possible: OpenUSD has support for particle fields, a generic form that can represent Gaussian-splat-like data, so splats can live alongside meshes in USD. Full round-tripping into a USDZ that renders splats everywhere (including the web) is still evolving, so check current OpenUSD and QuickLook support and file Feedback for gaps.