GoKawiilis a reporter focusing on film, TV, and pop culture. Before The Verge, he wrote about comic books, labor, race, and more at io9 and Gizmodo for almost five years.
Though it feels unlikely that gen AI will ever be able to whip up a compelling movie whole cloth, Tribeca featured a number of films that demonstrated how human artists can leverage the technology in compelling ways.
While none of the AI-powered movies that screened at Tribeca were as terrible as the video slop companies like OpenAI and xAI have polluted the internet with, some of the projects were prime examples of why generative content tends to feel so lifeless compared to human-crafted art. Roar — an animated short produced by Illuminai Studios — felt more like a disorienting montage of AI-generated clips rather than a cohesive piece of cinema. And Asteria Film Co.’s ChikaBOOM! lacked the visual and sonic polish that’s necessary for fast-paced fantasy about a magician in training to really pull you in.
Roar and ChikaBOOM!’s overall roughness seemed to be a reflection of the inherent technological limitations baked into their respective AI-forward production workflows. But other films, like Google DeepMind’s Dear Upstairs Neighbors and OpenAI’s Mauvais Soleil, showcased how it’s possible for filmmakers to avoid those challenges when gen AI is deployed with a bit more ingenuity.
Written and directed by Pixar veteran Connie Qin He in collaboration with researchers from Google DeepMind, Dear Upstairs Neighbors tells the story of an exhausted young woman who’s trying to go to bed. All Ada (Márcia Mayer, who also produced the short) wants is to get a couple hours of peaceful rest before she has to wake up and get back to work. But every time she begins dozing off, the cacophony of noise coming from her upstairs neighbors’ apartment jolts her awake and leaves her wondering what they could possibly be doing in the middle of the night.
To give Dear Upstairs Neighbors’ world a distinct style, He enlisted Pixar production designer Yingzong Xin, who painted concept art in Photoshop and on paper using acrylics. Those illustrations’ expressionistic aesthetic was key to bringing Dear Upstairs Neighbors’ fantastical story to life, but it also presented a unique challenge to DeepMind’s researchers. With most AI video generation models, the illustrations’ painterly style would be difficult to turn into visually consistent footage. But DeepMind’s engineers developed custom versions of Veo and Imagen that were specifically designed to give Dear Upstairs Neighbors’ artists the ability to fine-tune their outputs.
You can see how filmmakers have to work around some of gen AI’s more typical limitations.
Because the customized models were trained on Xin’s concept art, they could consistently generate shots that adhered to He’s vision for the project. The text-to-video models were great at reproducing certain stylistic details, like the way sound is visualized when objects interact with one another. But to really build Dear Upstairs Neighbors’ scenes in a way that would tell a cohesive story, the short’s creative team had to do things a bit more traditionally. By creating rough animations with Autodesk Maya (the industry standard for 3D rigging and VFX), Dear Upstairs Neighbors’ production team could ensure that scenes would unfold exactly how they wanted them to. And by feeding those roughs into Veo, the artists could create scenes that were more visually polished and ready to be further enhanced with additional stylized assets generated with Veo and Imagen.
More than any other film at Tribeca, Dear Upstairs Neighbors felt like a case study in how generative AI can be used as a bespoke tool that actually assists artists as they develop their ideas. The film’s entire workflow relied on human-made art and people making the kinds of nuanced creative decisions that text-to-video generators aren’t capable of on their own. It’s important to bear in mind that Dear Upstairs Neighbors would not be nearly as visually impressive if it had been produced with vanilla versions of Google’s various models. The models worked well for this particular short, but that’s to be expected for a project that’s also very much a commercial for Google’s technology.
Image: Google DeepMind
... continue reading