Developers can ship Oculus Quest apps or updates with mixed reality from the next SDK version. Spatial Anchors are coming soon, and Scene Capture will arrive next year. Facebook calls these capabilities the Insight SDK.
Basic mixed reality functionality has been available to developers as the Experimental Passthrough API since August. But apps on the Oculus Store or App Lab weren’t allowed to use it, restricting distribution to third party methods like SideQuest. With the next SDK update the Passthrough API will no longer be experimental and will be permitted on the Store and App Lab.
The API lets developers use passthrough – the view from the Quest’s greyscale tracking cameras – as a layer (eg. the background) or on a custom mesh (eg. a desk in front of you). Since black & white isn’t particularly appealing, passthrough can be stylized with edge rendering, a color overlay, or posterization.
Coming soon, Insight SDK will get experimental support for Spatial Anchors. These world-locked reference frames will let apps place content in a specific position users mark in their room, and the headset will remember these anchor positions between sessions.
Next year, Facebook plans to add Scene Understanding as an experimental feature. Don’t be fooled by that name though, this isn’t an automatic system. Users will be prompted to mark out their walls and furniture, and to enter their ceiling height – a process called Scene Capture. This needs to be done for each room but the headset should remember the Scene Model between sessions.
With Spatial Anchors and Scene Understanding, Facebook says developers will be able to attach a virtual screen to a specific place on your wall, show your real furniture in VR, or have a character walk around your room with realistic occlusion. In the short term however mixed reality functionality will be much more basic, and future hardware with color cameras will be needed to make the experience truly compelling.
