Imagine a busy street for just a few moments after which attempting to sketch the scene you saw from memory. Most people could draw the rough locations of key objects like cars, people, and crosswalks, but almost nobody can draw every detail with pixel-perfect accuracy. The same is true of most up-to-date computer vision algorithms: they’re excellent at capturing fine-grained details of a scene, but lose fine-grained details when processing information.
Now MIT researchers have developed a system called “FeatUp“This allows algorithms to capture all of the high- and low-level details of a scene concurrently – almost like Lasik eye surgery for computer vision.
As computers learn to “see” by images and videos, they develop “ideas” about what’s contained in a scene through so-called “features.” To create these features, deep networks and visual foundation models break down images right into a grid of tiny squares and process these squares as a bunch to find out what's happening in a photograph. Each tiny square typically consists of 16 to 32 pixels, so the resolution of those algorithms is significantly lower than the pictures they work with. When attempting to summarize and understand photos, algorithms lose lots of pixel sharpness.
The FeatUp algorithm can stop this information loss and increase the resolution of any deep network without compromising speed or quality. This allows researchers to quickly and simply improve the resolution of recent or existing algorithms. For example, imagine attempting to interpret the predictions of a lung cancer detection algorithm to locate the tumor. Applying FeatUp prior to interpreting the algorithm using a technique similar to Class Activation Maps (CAM) can provide a rather more detailed (16-32x) view of where the tumor could also be situated in response to the model.
In addition to helping practitioners understand their models, FeatUp can improve a lot of different tasks similar to object detection, semantic segmentation (assigning labels to pixels in a picture with object labels), and depth estimation. This is achieved by providing more accurate, high-resolution capabilities, that are critical to developing machine vision applications starting from autonomous driving to medical imaging.
“The essence of all computer vision lies in these deep, intelligent capabilities that emerge from the depths of deep learning architectures. The big challenge of recent algorithms is that they reduce large images to very small grids of “smart” features, gaining intelligent insights but losing the finer details,” says Mark Hamilton, an MIT doctoral candidate in electrical engineering and computer science at MIT Computer Science and Affiliate of the Artificial Intelligence Laboratory (CSAIL) and co-lead creator at a Paper in regards to the project. “FeatUp helps enable the very best of each worlds: very smart representations on the resolution of the unique image. These high-resolution capabilities significantly increase performance across a spectrum of computer vision tasks, from improving object detection and improving depth prediction to providing a deeper understanding of your network’s decision-making process through high-resolution evaluation.”
Renaissance of dissolution
As these large AI models turn out to be more widespread, there’s an increasing need to clarify what they do, what they appear at, and what they think.
But how exactly can FeatUp discover these fine-grain details? Curiously, the key lies within the shaking and wobbling of the pictures.
Specifically, FeatUp makes minor adjustments (similar to moving the image just a few pixels to the left or right) and observes how an algorithm responds to those minor movements of the image. This ends in lots of of deep feature maps, each barely different, that may be combined right into a single crisp, high-resolution set of deep features. “We imagine that there are some high-resolution features and that if we shake and blur them, they’ll match all the unique, lower-resolution features of the blurred images.” Our goal is to make use of this “game” to learn the way “we are able to refine the low-resolution features into high-resolution features so we understand how well we’re doing,” says Hamilton. This methodology is comparable to the best way algorithms can create a 3D model from multiple 2D images by ensuring that the anticipated 3D object matches all the 2D photos used to create it. In the case of FeatUp, they predict a high-resolution feature map that matches any low-resolution feature maps created by jittering the unique image.
The team found that the usual tools available in PyTorch weren’t sufficient for his or her needs, so seeking a quick and efficient solution, they introduced a brand new form of deep network layer. Their custom layer, a special joint bilateral upsampling operation, was over 100 times more efficient than an easy implementation in PyTorch. The team also showed that this latest layer can improve quite a lot of different algorithms, including semantic segmentation and depth prediction. This layer improved the network's ability to process and understand high-resolution details, providing a big performance boost to any algorithm that used it.
“Another application is so-called small object retrieval, during which our algorithm enables precise localization of objects. For example, even in cluttered street scenes, FeatUp-enriched algorithms can detect tiny objects like traffic cones, reflectors, lights, and potholes where their low-resolution cousins fail. This demonstrates its ability to convert coarse features into finely detailed signals,” says Stephanie Fu '22, MNG '23, a doctoral student on the University of California, Berkeley and one other co-lead creator of the brand new FeatUp paper. “This is especially necessary for time-critical tasks, similar to locating a traffic sign on a crowded highway in a self-driving automobile. Not only can this improve the accuracy of such tasks by turning rough guesses into accurate localizations, but it surely could also make these systems more reliable, interpretable and trustworthy.”
What next?
Looking to future endeavors, the team emphasizes FeatUp's potentially broad adoption inside the research community and beyond, just like data augmentation practices. “The goal is to make this method a fundamental tool for deep learning and enrich models to perceive the world in greater detail, without the computational inefficiency of traditional high-resolution processing,” says Fu.
“FeatUp represents a beautiful advance in making visual representations truly useful by producing them at full image resolution,” says Noah Snavely, a pc science professor at Cornell University who was not involved within the research. “Learned visual representations have gotten really good in recent times, but they’re almost all the time created at very low resolution – you would insert a pleasant full-resolution photo and what you get is a tiny, postage stamp-sized grid of features.” That's an issue if you desire to use these features in applications that produce full resolution output. FeatUp creatively solves this problem by combining classic super-resolution ideas with modern learning approaches, leading to beautiful, high-resolution feature maps.”
“We hope that this straightforward idea can find widespread application. “It provides high-resolution versions of image evaluation that we previously thought could only be low-resolution,” says senior creator William T. Freeman, MIT professor of electrical engineering and computer science and CSAIL member.
Lead authors Fu and Hamilton are joined by MIT graduate students Laura Brandt SM '21 and Axel Feldmann SM '21, and Zhoutong Zhang SM '21, PhD '22, all current or former MIT CSAIL employees. Her research is supported partly by a Graduate Research Fellowship from the National Science Foundation, from the National Science Foundation and the Office of the Director of National Intelligence, the US Air Force Research Laboratory, and the US Air Force Artificial Intelligence Accelerator. The group will present their work on the International Conference on Learning Representations in May.
