Meta's Wearables Device Access Toolkit, coming later this year, will let smartphone apps interact with the company's smart glasses.
With the initial release of the SDK, Meta says mobile developers will be able to access the camera, speakers, and microphone array of its full glasses lineup. Users will have to give each app permission to access their glasses.
Developers could, for example, leverage the SDK to add first-person livestreaming or recording features to their apps. Or they could feed the camera imagery to a third-party multimodal AI model to analyze what you're looking at.
At launch, Meta Wearables Device Access Toolkit won't support Meta AI. That's not to say that developers won't be able to build AI experiences. But they will have to initiate a continuous audio stream to and from the glasses and send camera images to their own AI model of choice, at their own cost, and this will impact battery life. Meta says adding Meta AI integration is a "key area" it's exploring for future updates to the toolkit.
The SDK also won't support sending imagery to the Meta Ray-Ban Display HUD, nor accessing gestures from its Meta Neural Band, but Meta says it's starting to think about how this could eventually work.
Interested developers can sign up for the Meta Wearables Device Access Toolkit preview by filling in this form.
Early Developer Experiments
Meta provided an early version of the Wearables Device Access Toolkit to a handful of developers several months ago, including Twitch, Microsoft, Logitech Streamlabs, and Disney.
Twitch and Logitech Streamlabs are using the SDK to let you livestream your first-person view on their platforms, just as you already can on Instagram, while Microsoft is using it for its Seeing AI platform that helps blind people navigate and interact with the world around them.
How 18Birdies is using the toolkit.
One particularly interesting use case comes from 18Birdies. The golf app is experimenting with using Meta Wearables Device Access Toolkit for real-time yardages and club recommendations, helping golfers without requiring them to take their phone out of their pocket.
Another is from Disney's Imagineering team, who are exploring using the toolkit to give guests a personal AI guide in Disney parks.
David Heaney
