Creating realistic 3D models for applications corresponding to virtual reality, filmmaking, and technical design is usually a laborious process that requires plenty of manual trial and error.
While generative artificial intelligence models for images can streamline artistic processes by allowing creators to create lifelike 2D images from text prompts, these models usually are not designed to generate 3D shapes. To fill this gap, a recently developed technique called Score distillation uses 2D imaging models to create 3D shapes, however the output is commonly blurry or cartoonish.
MIT researchers examined the relationships and differences between the algorithms used to generate 2D images and 3D shapes and identified the foundation explanation for lower quality 3D models. From there, they developed an easy rating distillation solution that permits the generation of sharp, high-quality 3D shapes which might be closer in quality to one of the best model-generated 2D images.
Some other methods attempt to deal with this issue by retraining or fine-tuning the generative AI model, which will be expensive and time-consuming.
In contrast, the MIT researchers' technique achieves 3D shape quality equal to and even superior to those approaches without additional training or complex post-processing.
Additionally, by identifying the explanation for the issue, researchers have improved the mathematical understanding of rating distillation and related techniques, enabling future work to further improve performance.
“Now we all know where to go, which allows us to search out more efficient solutions which might be faster and of upper quality,” says Artem Lukoianov, a doctoral student in electrical engineering and computer science (EECS) and lead writer of a paper with regards to this method . “In the long run, our work might help make the method easier as a co-pilot for designers and make it easier to create more realistic 3D shapes.”
Lukoianov's co-authors are Haitz Sáez de Ocáriz Borde, a graduate student at Oxford University; Kristjan Greenewald, research scientist within the MIT-IBM Watson AI Lab; Vitor Campagnolo Guizilini, scientist at Toyota Research Institute; Timur Bagautdinov, research scientist at Meta; and senior authors Vincent Sitzmann, assistant professor of EECS at MIT, who leads the Scene Representation Group within the Computer Science and Artificial Intelligence Laboratory (CSAIL), and Justin Solomon, associate professor of EECS and leader of the CSAIL Geometric Computing Group. The research might be presented on the Conference on Neural Information Processing Systems.
From 2D images to 3D shapes
Diffusion models like DALL-E are a variety of generative AI model that may generate lifelike images from random noise. To train these models, researchers add noise to the pictures after which teach the model to reverse the method and take away the noise. The models use this learned “denoising” process to create images based on a user’s text prompts.
However, diffusion models are poorly acting at directly generating realistic 3D shapes because there will not be enough 3D data to coach them. To get around this problem, researchers developed a method called Score distillation sampling (SDS) in 2022, which uses a pre-trained diffusion model to mix 2D images right into a 3D representation.
The technique is to start out with a random 3D representation, render a 2D view of a desired object from a random camera angle, add noise to that image, denoise it with a diffusion model, after which optimize the random 3D representation in order that that it matches the denoised image. These steps are repeated until the specified 3D object is generated.
However, 3D shapes created this fashion are inclined to appear blurry or oversaturated.
“This has been a bottleneck for a while. We know that the underlying model can produce higher results, but people didn’t know why this happens with 3D shapes,” says Lukoianov.
The MIT researchers examined the steps of SDS and identified a discrepancy between a formula that represents a key component of the method and its counterpart in 2D diffusion models. The formula tells the model update the random representation, adding and removing noise step-by-step to make it more much like the specified image.
Because a part of this formula involves an equation that is simply too complex to resolve efficiently, SDS replaces it with randomly sampled noise at each step. The MIT researchers found that this noise leads to blurry or cartoon-like 3D shapes.
An approximate answer
Instead of trying to resolve this cumbersome formula exactly, the researchers tested approximate techniques until they found one of the best solution. Instead of randomly sampling the noise term, their approximation technique derives the missing term from the present 3D shape representation.
“In this fashion, because the evaluation within the paper predicts, 3D shapes are created that look sharp and realistic,” he says.
In addition, the researchers increased the resolution of the image rendering and adjusted some model parameters to further improve the standard of the 3D shapes.
In the top, they were in a position to use an off-the-shelf, pre-trained image diffusion model to create smooth, realistic-looking 3D shapes without the necessity for costly retraining. The 3D objects are similarly sharp to those produced using other methods based on ad hoc solutions.
“Trying to experiment blindly with different parameters, sometimes it really works and sometimes it doesn't, but you don't know why. We know that is the equation we want to resolve. This now enables us to search out more efficient solutions,” he says.
Because their method relies on a pre-trained diffusion model, it inherits the biases and shortcomings of that model, making it liable to hallucinations and other errors. Improving the underlying diffusion model would improve their process.
In addition to studying the formula to see how they may solve it more effectively, researchers are thinking about studying how these findings could improve image editing techniques.
This work is funded partially by the Toyota Research Institute, the US National Science Foundation, the Singapore Defense Science and Technology Agency, the US Intelligence Advanced Research Projects Activity, the Amazon Science Hub, IBM, the US Army Research Office, the CSAIL Future of Data Program, Wistron Corporation and the MIT-IBM Watson AI Laboratory.
