Hey HN, I’m Jash, cofounder of Flywheel AI ( https://useflywheel.ai ) and formerly https://remoteteleop.com . My cofounder, Mahimana Bhatt, built ML data and eval pipeline at autonomous driving company Motional. We’re building remote teleop and autonomous stack for excavators.
Interfacing with existing excavators for enabling remote teleop (or autonomy) is hard. Unlike cars which use drive-by-wire technology, most of the millions of excavators are fully hydraulic machines. The joysticks are connected to a pilot hydraulic circuit, which proportionally moves the cylinders in the main hydraulic circuit which ultimately moves the excavator joints. This means excavators mostly do not have an electronic component to control the joints. We solve this by mechanically actuating the joysticks and pedals inside the excavators.
We do this with retrofits which work on any excavator model/make, enabling us to augment existing machines. By enabling remote teleoperation, we are able to increase site safety, productivity and also cost efficiency.
(Remote) Teleoperation by the operators enables us to prepare training data for autonomy. In robotics, training data comprises observation and action. While images and videos are abundant on the internet, egocentric (pov) observation and action data is extremely scarce, and it is this scarcity that is holding back scaling robot learning policies.
Flywheel solves this by preparing the training data coming from our remote teleop-enabled excavators which we have already deployed. And we do this with very minimal hardware setup and resources.
During our time in YC, we did 25-30 iterations of sensor stack and placement permutations/combinations, and model hyperparams variations. We called this “evolution of the physical form of our retrofit”. Eventually, we landed to our current evolution and have successfully been able to train some levels of autonomy with only a few hours of training data. We will continue playing around with these ‘knobs’ to figure out what works better.
The big takeaway was how much more important data is than optimizing hyperparams of the model. So today, we’re open sourcing 100hrs of excavator dataset that we collected using Flywheel systems on real construction sites. This is in partnership with Frodobots.ai
Some details here: https://youtu.be/zCNmNm3lQGk
Machine/retrofit details: Volvo EC380 (38 ton excavator) 4xcamera (25fps) 25 hz expert operator’s action data
We’re just getting started. We have good amounts of variations in daylight, weather, tasks, and would be adding more hours of data and also converting to lerobot format soon. We’re doing this so people like you and me can try out training models on real world data which is very, very hard to get.
So please checkout the dataset here and feel free to download and use however you like. We can’t wait to see what you all build. I’ll be around in the thread and look forward to comments and feedback from the community!
Dataset link: https://huggingface.co/datasets/FlywheelAI/excavator-dataset