GoKawiilAn early version of FLUX 3, our new multimodal foundation model, is now running on robots. We gave mimic robotics early access to FLUX.3. Their strength in robot learning and deployment, combined with the model's world knowledge and BFL's foundation model expertise, produced FLUX-mimic: the next generation of video-action models.
FLUX
FLUX 1 and FLUX 2 generate images. FLUX 3 expands into multimodality and generates audio-visual content jointly - and, at the same time, provides the foundation of FLUX-mimic: A video-action model, developed in collaboration with mimic, running robots that have been tested and deployed at Audi.
At first glance, producing convincing visual content and controlling robots seem to have little in common. One requires generating pixels, the other an understanding of how the physical world responds when you touch and manipulate it. If one model does both, it was never really only a content creation model. It is a model of how the world behaves, and content creation is one thing one can do with it.
That is what FLUX 3 is.
Video is the hard part
FLUX 3 is one model, jointly trained across images, video and audio from the beginning. The most demanding part of that training - accounting for over 95% of the total compute costs - is video prediction. To generate realistic videos, a model has no choice but to learn contact, motion, weight, cause and effect; get any of them wrong and it looks wrong. Learning to render the world accurately means learning how the world behaves.
Relatively speaking, audio is the easy modality. Low dimensional and far less detailed than video, it makes up less than 0.5% of the tokens in a 720p video with audio. Once a model has done the hard work of learning video understanding, it will learn the causal relationship between video and audio to predict speech synchronized to lip movement and audio effects synchronized to the physical events causing them.
Actions follow the same shape: a low dimensional representation of a robot's state, tightly coupled to visual observations. Actions, audio and video frames are all partial representations of a single underlying physical reality. After the model has learned about the physical processes behind video and audio, action prediction is not a new departure - it is one more view of the reality it already models.
A single backbone
... continue reading