Bringing 3D shoppable products online with generative AI

Second generation: Scaling with a view-conditioned diffusion prior In 2023, we introduced a second-generation approach which used a view-conditioned diffusion prior to address the limitations of the first approach. Being view-conditioned means that you can give it an image of the top of a shoe and ask the model “what does the front of this shoe look like?” In this way, we can use the view-conditioned diffusion model to help predict what the shoe looks like from any viewpoint, even if we only have photos of limited viewpoints. In practice, we employ a variant of score distillation sampling (SDS), first proposed in DreamFusion. During training, we render the 3D model from a random camera view. We then use the view-conditioned diffusion model and the available posed images to generate a target from the same camera view. Finally, we calculate a score by comparing the rendered image and the generated target. This score directly informs the optimization process, refining the 3D model's par ... Read full article.