GoKawiilAt Pantograph, we're working on training fully general robotics models that can act autonomously for hours at a time. Especially in robotics, it's difficult to get diverse data at scale. Learning to act from internet video data could allow models to scale with compute, rather than being limited by small action datasets.
In this work, we develop a simple method for learning goal-directed behavior through pretraining on internet-scale video.
Usually, goal-directedness is taught in a post-training phase, which limits the extent to which it can generalize. Here, we learn goal-directedness during pretraining on internet-scale video, which greatly improves the models' ability to achieve complex goals.
Video games are a useful testing ground for robotics. We're starting with Minecraft because it's open-ended, and supports commanding diverse, long-horizon goals.1This model can only act within the one game, but we will be training larger models across a broader distribution of visual environments in the coming months.↩ Our largest model, which we call Pan, is a 4B parameter model that can fight mobs, explore to find specific objects, complete challenging platforming environments, and build structures as commanded. We find it is able to generalize to quite diverse environments that it has never seen before.
Reinforcement learning is interesting in part because it can produce models that are superhuman. If we can do reinforcement learning at scale on video, we may be able to produce models that are superhuman in many more areas all at once.
#Reinforcement learning from observation-only data
We view internet-scale videos as reinforcement learning trajectories that include only observations.2This lens makes sense for some modalities and not others. Most text data, for example, can't be viewed as a written transcript of an agent interacting with an environment. It could likely work for audio, however.↩ It's difficult to do reinforcement learning from this kind of data because it lacks rewards and actions, but it turns out there are simple solutions for both.
Goal-conditioning is a simple way of side-stepping the need to specify a reward function. The idea is to use what happens later in a video as the goal for the earlier part, a trick sometimes called hindsight relabeling,3Marcin Andrychowicz et al., Hindsight Experience Replay, NeurIPS 2017.↩ because it takes what may have been a failure at one goal, and relabels it as a success for whatever actually happens. In principle, you could use any description of what happens later in a video as the goal, but in this work we focus on using frames of video as goals.
Although videos don't contain actions, it's still possible to learn functions from video that only depend on states. This includes value functions,4Typically, value functions are tied to a specific policy. In this very general setting, where any number of policies may be acting in any kind of environment, it makes more sense to use in-context learning to infer the tendencies and level of skill of the policies in a specific video. This way a single model can be trained that can act as a generic value function, capable of evaluating any policy's behavior.↩ which in goal-conditioned RL represent roughly the probability that a goal frame is achieved in the agent's future, as well as the next frame distribution for a goal-conditioned policy.5Both of these can be modeled in a number of ways, including likelihood-based models, energy-based models, and contrastive learning. We're excited about future work that could study the trade-offs between these different approaches.↩
After this action-agnostic pretraining, a smaller dataset containing actions can be used to produce an agent capable of acting in the environment.6Given a strong value function, learning to act is a much easier problem (Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018). A value function effectively converts the very difficult multi-step decision making problem of reinforcement learning into a single-step decision problem, which is generally much more tractable.↩
... continue reading