visionOS 2 brings major improvements to hand tracking on Apple Vision Pro and expands the headset's understanding of the environment around you.
David Heaney
These improvements weren't announced in the main WWDC24 keynote, but Apple engineers detailed them in developer sessions available online.
Hand Tracking Improvements
In our review of Apple Vision Pro we criticized the fact that developers received a very low update-rate for hand tracking, making games like Super Fruit Ninja and Synth Riders feel sluggish and unresponsive.
With visionOS 2 Apple is bringing two significant improvements to hand tracking.
The first improvement is increased update rate. On visionOS 1 the system provided developers with hand tracking poses at 30Hz. On visionOS 2 hand tracking poses are provided at the display refresh rate, typically 90Hz.
The other improvement is prediction, which is optional for developers. Enabling prediction will provide extrapolated hand poses representing where the hand should be by the time each frame is displayed. This results in noticeably lower effective latency, but with the tradeoff of some accuracy.
Apple recommends developers use prediction to attach virtual objects to your hands, but not when fluidity is important such as 3D drawing where you want the lines to be smooth.
Scene Understanding Improvements
Since launch Apple Vision Pro has provided developers with a continuously updated 3D mesh of the environment, generated by the LiDAR sensor. But building useful apps that take advantage of this requires understanding what exists within this mesh.
With visionOS 2 the headset now has Room Tracking. The system will provide developers with precise geometry of the walls and floor of each room, even supporting non-rectangular walls.
When entering a new room, the system will inform the developer and start providing geometry only for the new room. Apple says a developer could use this to have a virtual pet greet you when you enter your bedroom, for example.
Beyond just walls and floors, visionOS 2 also brings improvements to plane detection.
While visionOS 1 could only detect horizontal and vertical planes, visionOS 2 can detect slanted planes too.
These improvements should allow visionOS developers to build apps more aware of your physical environment that adapt to the space you're in and anchor to real world surfaces.
visionOS 2 also adds the ability to track 3D objects, an entirely new tracking capability.
David Heaney
The first visionOS 2 beta is available to install already, and the stable release is set for fall. Expect to see App Store apps that make use of these improvements later this year.
