New Apple study teaches robots how to act by watching first-person videos of humans

In a new paper called “Humanoid Policy ∼ Human Policy,” Apple researchers propose an interesting way to train humanoid robots. And it involves wearing an Apple Vision Pro. Robot see, robot do The project is a collaboration between Apple, MIT, Carnegie Mellon, the University of Washington, and UC San Diego. It explores how first-person footage of people manipulating objects can be used to train general-purpose robot models. In total, the researchers gathered over 25,000 human demonstrations and 1,500 robot demonstrations (a dataset they called PH2D), and fed them into a unified AI policy that could then control a real humanoid robot in the physical world. As the authors explain: Training manipulation policies for humanoid robots with diverse data enhances their robustness and generalization across tasks and platforms. However, learning solely from robot demonstrations is labor-intensive and requires expensive teleoperated data collection, which is difficult to scale. This paper inv ... Read full article.