GoKawiilIn my office, there are bookshelves where I keep my magic tricks. Next to them are panoramic windows looking out on Paris -- except, not really. They're mixed reality overlays that greet me when I wear Apple's Vision Pro headset, mapped to my actual windows. A virtual clock hangs on the wall.
The bookshelves are real; the Paris windows and wall clock aren't. But Siri, which I invoke as a glowing orb in front of me, sees them all. When I ask what's in front of me, Siri's text box describes it all.
My first little moments wearing the Vision Pro and testing an all-seeing Siri AI in the developer preview of VisionOS 27 are surprising but very familiar. I've had moments like this in Samsung's Galaxy XR headset, which already has a Gemini Live mode that can recognize your room and the virtual apps and windows that are open. I've also worn many smart glasses where I've asked camera-aware AI to tell me about the world as I've walked around.
As I expected, Apple's got a new supercharged Siri coming to a range of devices this fall, from Watches to iPhones to iPads, and is expanding its visual intelligence capabilities beyond the iPhone. The Vision Pro's Siri is the most fascinating, though, and maybe the most ambitious. It can see everything in your field of view on command, like a sensory companion.
Not everything is perfect in this very early preview build, revealed just days ago at Apple's WWDC, but it already works well enough to show what could come next. Yes, glasses that could do this feel entirely possible now. In the meantime, the Vision Pro's assistive functions are going to get pretty interesting, too.
Siri knows my bookshelf, and my Paris virtual window, and my virtual clock. Screenshot by Scott Stein/CNET
Siri's glowing orb is ready to see from the get-go
Unlike the Visual Intelligence modes on iOS and iPadOS, which launch via the Camera app, the Vision Pro's Siri can see things when you simply say, "Hey, Siri," and ask a question.
Siri appears as a 3D glowing orb, manifesting in my room like a spirit. You can drag and drop it anywhere, but it's neat to see the glowing marble cast light effects on my desk or the room using VisionOS' spatial graphics engine.
I was able to say, "Hey, Siri, what's in front of me?" and it triggered the Visual Intelligence, making a new-sounding chirp and then flashing a quick visual scan around the area my eyes were looking, using eye tracking. It casts a wide net: It could see my bookshelf full of books in front of me and all the little action figures and toys I put there. It read a few book titles back to me, like Uzumaki and Wonderbook.
... continue reading