GoKawiilThe frontier of physical AI is a Jenga game in a warehouse in San Leandro, California.
That warehouse is occupied by Encord, a company that builds data tooling used to train AI models. Andrew Ceja is a pilot—the company’s term for its robotic trainers—and he’s carefully pulling wooden blocks from a tottering tower while wearing a headset with a camera that tracks what he sees. That alone is fairly common for collecting robot training data, but this headset includes sensors that measure his brain waves as he carefully disassembles the block tower.
Encord is one of a small but growing number of startups betting the next real constraint on humanoid and warehouse robotics won’t be model architecture but instead the sheer scarcity of real-world physical training data. Rather than just helping robotics companies manage the data they have, Encord is building a business around manufacturing the data they don’t.
The brain wave headset Ceja is wearing was built by Zander Labs, a German neuroscience startup that’s betting measuring brain activity — to deduce mental states like error, intent and surprise — can create a more useful data set to train models. Encord’s work with Zander is currently a trial run; Encord says the goal is to build an initial brain wave-tagged data set, run it through customer robotics models, and evaluate whether it actually improves performance before deciding whether to scale it up.
Lucas Gehrke, a Zander neuroscientist supervising the work, says that the amount of brain activity used at any point during a given task offers clues for model builders trying to figure out when they need to deploy their highest-effort models.
This is the “bleeding edge” of the effort to solve the robotics data bottleneck, according to Vineeth Velmurugan, Encord’s head of robot learning. A veteran of OpenAI’s robot lab and Berkshire Grey, the warehouse automation firm, Velmurugan joined Encord to build the company’s internal data-creation team.
Encord was founded to help companies building machine-vision applications annotate data and evaluate models. As their customers—Velmurugan says they work with many leading robotics firms but that he’s not authorized to name them—began to apply end-to-end learning to robotic manipulation tasks, executives realized they would have to produce training data themselves, rather than simply manage it. “The data simply does not exist,” Velmurugan said.
The bet that generative AI can do for robots what it’s done for chatbots keeps running into this same wall. Self-driving car companies collect physical-world data themselves, but that’s hard to scale. Training from video can work, but it lacks the fidelity of real world data. Velmurugan says it will take a data set something like five times the size of YouTube’s video corpus to break through—a scale that helps explain why data-generation itself has become a business and not just a research problem.
Feed your egocentric data needs
Companies building robot brains are now turning to two main sources: “egocentric” video collected by workers wearing cameras, often augmented with additional camera angles and other metrics, and data from robots operated remotely. Encord does both, drawing egocentric data from several factories around the globe, and using its San Leandro facility to experiment with new modalities, like brain waves, or collect data sets around specific skills for fine-tuning.
... continue reading