DreamFusion

DreamFusion is a product that uses a text-to-image generative model called Imagen to optimize a 3D scene.

The product features Score Distillation Sampling (SDS), a way to generate samples from a diffusion model by optimizing a loss function. SDS allows for the optimization of samples in an arbitrary parameter space, such as a 3D space.

The product uses a 3D scene parameterization similar to Neural Radiance Fields to define this differentiable mapping. DreamFusion adds additional regularizers and optimization strategies to improve geometry, resulting in coherent, high-quality normals, surface geometry and depth, and relightable NeRFs with a Lambertian shading model.

The use cases for DreamFusion involve optimizing 3D scenes for various applications, such as video games, virtual reality experiences, and film production.