GoKawiilPART 3
Dynamics Self-Guidance
Zero-shotting Rewards
Now the actual co-design problem: how do we generate a high-reward robot for a reward function the model has never seen during training? Here is the trick. The same model predicts both the embodiment and its dynamics, so we can feed those predictions into the user's reward function and get a predicted reward. With predicted rewards in hand, we can run the model many times in parallel, predict the reward for each candidate, and return the best one. That is best-of-N sampling, but at GPU speed. We can do better. Because the predicted reward is a differentiable function of the embodiment tokens, we can take its gradient and inject it back into the diffusion process — pushing samples toward higher reward at every denoising step, following classifier-guided DDIM11. The model is, in effect, asking itself: “how should this robot change to increase the reward?” We call this Dynamics Self-Guidance.
Why “self-guidance”: related work that incorporates dynamics into diffusion guidance trains a separate dynamics model12 or relies on a differentiable simulator13. Here, the model uses its own dynamics predictions to steer itself. related work that incorporates dynamics into diffusion guidance trains a separate dynamics modelor relies on a differentiable simulator. Here, the model uses its own dynamics predictions to steer itself.
One gradient, the whole episode. Transformer Transformer predicts state and action tokens across the episode's horizon in parallel; the reward gradient then flows back through all of it to the embodiment tokens. Hardware gets optimized over long-horizon dynamics, not one timestep at a time.
Reward in, robot out
Take a randomly-generated quadruped that falls during a dynamic toss. Ask our model to generate a robot for the same trajectory with only a tracking reward and it lowers the center of mass and widens the support polygon — stable design choices it discovered as a side-effect of trying to maximize tracking reward it has never seen. Now add a torque-penalty term and the optimization landscape shifts. The model returns a smaller, lighter robot — still tracking the motion, but using 30% less average torque. Add a size penalty and the landscape shifts again. This is what people might call zero-shotting rewards: train on dynamics modeling, optimize at inference for any reward you can write down.
Same model, different conditioning, different robots. Swapping the target trajectory (Floor Scrub → Dynamic Toss) or adding a size penalty reshapes the optimization landscape, and the generated embodiments shift accordingly — without retraining. Discrete choices (leg design, DoF) and continuous choices (link length, mounting point) both vary in response.
Composing diffusion models for whole datasets
... continue reading