Meta's research department has introduced SAM 2 (Segment Anything Model 2), an AI system that represents a major advancement in video analytics.
This latest model extends the image segmentation capabilities of its predecessor, SAM, and ventures into the more complex realm of video.
Video segmentation – the power to discover and track specific objects in a moving scene – has long been a challenge for AI.
While humans can effortlessly follow a automotive moving through traffic or an individual walking through a crowd, AI systems often have difficulty doing so.
This is a serious problem for self-driving cars and other autonomous vehicles (AVs) that have to track moving 3D objects of their environment.
SAM2 goals to shut this gap and produce AI's video understanding closer to human perception.
The system can discover and track virtually any object in a video with minimal user input – sometimes only a single click.
This opens up a world of possibilities in areas starting from film editing to scientific research.
This is how Meta created SAM 2:
- The team developed a method called promptable visual segmentation (PVS), which allows users to guide the AI with easy cues in each video frame. This means the system can adapt to quite a lot of scenarios, from tracking a selected person in a crowd to tracking the wing movement of a bird in flight.
- They built a model architecture that included components for processing individual frames, storing details about objects over time, and generating precise segmentations. A key element is the memory module, which allows SAM 2 to keep up consistent tracking even when objects temporarily disappear from view.
- An enormous latest dataset was created containing over 50,000 videos and 35 million labeled frames, dwarfing previous video segmentation datasets. This dataset, called SA-V, covers a big selection of object types, sizes, and scenarios, improving the model's ability to generalize to latest situations.
- The model was extensively trained and tested on 17 different video datasets, starting from dashcam footage to medical images. SAM 2 outperformed state-of-the-art methods on semi-supervised video object segmentation tasks, achieving a median improvement of seven.5% in J&F scores (a typical metric for segmentation quality).
Over: Image segmentation for complex video clips separates different shapes in seconds.
- In film production, SAM 2 could streamline visual effects work and save time in post-production
- Scientists could track cells in microscopic images or monitor environmental changes in satellite images
- For AVs, including self-driving cars, SAM 2 could improve object detection in complex traffic scenarios
- Conservationists could use SAM 2 to watch animal populations across vast areas
- In AR/VR, it might probably enable more accurate interactions with virtual objects in live videos
True to Meta's commitment to open research, SAM 2 is released as open source software.
This includes not only the model, but in addition the dataset used for training.
Researchers are already investigating ways to process longer videos, improve performance at tremendous details, and reduce the computational power required to run the model.
As image segmentation technology advances, the way in which we interact with and analyze video content is certain to vary.
SAM 2 pushes the boundaries of visual manipulation by making complex editing tasks more accessible and enabling latest types of visual evaluation.
