OpenAI's Sora, which might generate videos and interactive 3D environments on the fly, is a remarkable demonstration of GenAI's excellence – a real milestone.
But strangely, one in every of the innovations that led to that is an AI model architecture colloquially generally known as a diffusion transformer. arrived on the AI research scene years ago.
The Diffusion Transformer, which also powers AI startup Stability AI's latest image generator, Stable Diffusion 3.0, appears poised to remodel the GenAI field by enabling GenAI models to scale beyond what was previously possible.
Saining Xie, a pc science professor at NYU, began the research project that produced the diffusion transformer in June 2022. With William Peebles, his mentee when Peebles interned at Meta's AI research lab and now Sora's co-lead at OpenAI, Xie combined two machine learning concepts – and that – to create the Diffusion Transformer.
Most modern AI-powered media generators, including OpenAI's DALL-E 3, depend on a process called diffusion to output images, videos, voice, music, 3D meshes, graphics, and more.
It's not probably the most intuitive idea, but essentially noise is slowly added to a medium – resembling a picture – until it becomes unrecognizable. This is repeated to create a loud media dataset. When trained on this, a diffusion model learns to progressively subtract the noise, progressively approaching a goal output medium (e.g. a brand new image).
Diffusion models typically have a “backbone” or some sort of engine, a so-called U-Net. The U-Net backbone learns to estimate the noise that should be removed – and it does it well. But U-Nets are complex and have specially designed modules that may dramatically decelerate the diffusion pipeline.
Fortunately, transformers can replace U-Nets – increasing efficiency and performance in the method.
A video created by Sora. Photo credit: OpenAI
Transformers are the architecture of selection for complex reasoning tasks and support models resembling GPT-4, Gemini and ChatGPT. They have several unique properties, however the defining feature of transformers is their “attention mechanism.” For each bit of input data (within the case of diffusion image noise), transforms the relevance of all other inputs (other noise in a picture) and uses them to generate the output (an estimate of image noise).
The attention mechanism not only makes Transformers simpler than other model architectures, but in addition makes the architecture parallelizable. In other words: increasingly larger transformer models will be trained with significant, but not unattainable, increases in computing power.
“What transformers contribute to the diffusion process is comparable to a motor upgrade,” Xie told TechCrunch in an email interview. “The introduction of transformers…marks a major leap in scalability and effectiveness. This is especially evident in models like Sora, which profit from training on massive amounts of video data and utilize extensive model parameters to exhibit the transformative potential of transformers when applied at scale.”
Generated by stable diffusion 3. Photo credit: Stability AI
Considering that the thought for diffusion transformers has been around for some time, why did it take years for projects like Sora and Stable Diffusion to start out using them? Xie believes that the importance of a scalable backbone model has only been recognized relatively recently.
“The Sora team really did their best to indicate how way more you possibly can achieve with this approach at scale,” he said. “They’ve made it pretty clear that U-Nets are out Transformers are imminent diffusion models any further.”
Diffusion transformers are a straightforward substitute for existing diffusion models, says Xie – no matter whether the models produce images, video, audio or one other type of media. The current means of training diffusion transformers may introduce some inefficiencies and lack of performance, but Xie believes this will be fixed in the long term.
“The key takeaway is pretty easy: Forget U-Nets and switch to transformers, because they’re faster, work higher and are more scalable,” he said. “I’m curious about integrating the areas of understanding and content creation throughout the framework of diffusion transformers. At the moment, these are like two different worlds – one for understanding and one other for creating. I envision a future where these features are integrated, and I consider that achieving this integration requires standardization of the underlying architectures, with transformers being a great candidate for this purpose.”
If Sora and Stable Diffusion 3.0 are a preview of what's to come back with Diffusion Transformers, I'd say we're in for a wild ride.
