GoKawiilPosts from this author will be added to your daily email digest and your homepage feed.
This is Lowpass by Janko Roettgers, a newsletter on the ever-evolving intersection of tech and entertainment, syndicated just for The Verge subscribers once a week.
When Mark Drummond was working on the Vision Pro at Apple, he had a bit of an epiphany that didn’t really fit Cupertino’s preferred narrative.
Drummond was managing the Character Intelligence Team, which among other things built the Encounter Dinosaurs demo. Preinstalled on the Vision Pro, the demo puts viewers eye to eye with interactive prehistoric creatures.
“We built that with Jon Favreau,” Drummond says, referring to Apple’s long-standing partnership with the Mandalorian director. “Before the headset was available to us to take down to Burbank, we used iPhones and iPads,” he recalls. Relying on mobile devices for demos made sense. VisionOS is essentially an iPadOS fork. “It did actually work out. We had a really good experience with iPhone and iPad,” he says.
Through that experience, Drummond realized that these mobile devices weren’t just decent stand-ins for the headset. “What we learned looking for sources of surprise and delight with interactive characters in mixed reality is that the headset is actually not the best [device] for this kind of thing,” he says.
“I still think it’s a pretty fabulous piece of hardware,” Drummond says. However, headsets can also be alienating, and separate viewers from the world and from the people around them. “It’s kind of lonely,” Drummond says. Having an AR app on a phone, on the other hand, makes it much easier to show it to others. “People can lean in over your shoulder,” Drummond says.
That’s why, shortly after leaving Apple in 2023, Drummond embraced mobile devices for augmented reality storytelling. For the past two and a half years, he has been working on a new iPhone AR app called Pixi that’s all about mobile-first interactive storytelling.
In short, Pixi is working on something that could best be described as the AR version of the email greeting card. When it launches in the coming weeks, Pixi will let anyone pick an interactive character and a scenario, add a personalized message, and then send it to their contacts via iMessage or WhatsApp. Once the recipient opens such a Pixi message, the character appears overlaid in the camera view of their real-world environment, and interacts with them.
During a recent demo, Pixi’s AR experiences included an animated cat and a robot. They could tell jokes, play tic-tac-toe with you, or challenge you to a game of whack-a-mole, right on your desk.
... continue reading