Adobe researchers introduced VideoGigaGAN, a generative AI model that may upscale blurry videos into crisp, smooth videos that look as much as 8x sharper.
We've had really good image upscalers for some time, but constructing a great video upscaler is exponentially tougher.
Video Super Resolution (VSR) takes individual frames of a video, upscales the resolution and details, and stitches the pictures together to recreate the video.
To do that well, two contradictory challenges have to be resolved. Current VSRs produce either smooth and blurry videos or sharp and glitchy videos.
Adobe's VideoGigaGAN upsamples blurry videos to supply video that’s each temporally consistent (smooth frame transitions) and high-frequency detail.
Here is an example of what VideoGigaGAN can do.
Adobe research drops VideoGigaGAN
It lets you upsample videos 8x with improved details.
Paper in comments 👇 pic.twitter.com/7uEiU7bYqw
— Kris Kashtanova (@icreatelife) April 22, 2024
As the name suggests, Adobe's method relies on GigaGAN, a sophisticated generative adversarial network (GAN).
GANs are great for upsampling images, and GigaGAN is among the finest at image super-resolution. So why not only use GigaGAN on each frame to upscale the image after which stitch all of them together to create the video?
When Adobe researchers tried to attain excellent video resolution, the resulting video was temporally inconsistent and flickered.
Adding temporal convolution and a focus layers to the GigaGAN resolved the temporal inconsistency, but flickering was still a difficulty.
VideoGigaGAN addresses this problem by separating low-frequency and high-frequency elements in each frame and processing them in another way.
The low-frequency feature map is smoothed to remove high-frequency details that could be sources of noise and flicker.
Skip connections preserve the finer details in high-frequency components by bypassing the center layers within the model that might otherwise be lost in processing.
You can read more in regards to the technical details here Adobe's article.
The Demos on Adobe's GitHub are very impressive. Adobe hasn't mentioned a release date, but let's hope we get to make use of it soon.
Imagine what a tool like this might do for historical archive footage, classic movies, and even upscaling your old favorite TV shows into HD.
