SAM 2 (Segment Anything Model 2 / 2.1)
Robotics modelApache-2.0 foundation model from Meta FAIR for real-time promptable visual segmentation and multi-object tracking across static images and videos using a Hiera backbone and streaming memory transformer. [1] [2] [5]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Robotics perception engineers and researchers building object segmentation, instance tracking, or open-vocabulary perception pipelines (e.g., Grounded SAM 2); Commercial and academic teams seeking Apache-2.0 image/video segmentation weights and fine-tuning recipes [1] [2]
- Parameters
- 38.9M (Hiera-Tiny), 46M (Hiera-Small), 80.8M (Hiera-Base+), 224.4M (Hiera-Large) [1]
- Source rights
- Named licenses are documented by part. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
Overview
SAM 2 (facebookresearch/sam2, arXiv:2408.00714) is a foundation visual segmentation model from Meta FAIR that extends Segment Anything from static images to real-time multi-object video segmentation using a streaming memory transformer. At pinned commit 2b90b9f5ceec907a1c18123530e92e794ad901a4 (December 2024), the repository ships the improved SAM 2.1 checkpoints, full-model torch.compile VOS acceleration (vos_optimized=True), independent per-object tracking in SAM2VideoPredictor, Hydra-based training/fine-tuning code, and a deployable web demo. [1] [3] [5] [6]
Model card summary
- Architecture
Hierarchical Vision Transformer (Hiera: Tiny 38.9M, Small 46M, Base+ 80.8M, Large 224.4M) image encoder coupled with a prompt encoder, mask decoder, and streaming memory attention bank enabling single-frame image segmentation and real-time multi-object video masklet propagation (with optional full-model torch.compile VOS optimization).
- Parameter count
38.9M (Hiera-Tiny), 46M (Hiera-Small), 80.8M (Hiera-Base+), 224.4M (Hiera-Large)
Exact parameter counts reported in README Model Description table for both SAM 2.1 and SAM 2 checkpoints.
- Input modalities
- rgb
Static RGB images or RGB video frames plus interactive spatial prompts (points, bounding boxes, or masks).
- Output modalities
- video-frames
Outputs per-frame binary segmentation masks and spatio-temporal multi-object video masklets rather than robot control commands.
- Action space
Perception-only visual segmentation model (no robot action output); produces 2D segmentation masks for images and tracked multi-object masklets across video frames.
- Trained / supported embodiments
- Embodiment-agnostic 2D RGB image and video segmentation foundation model
- Training datasets
- SA-V (Segment Anything Video: 50,583 train / 155 val / 150 test videos, 643K masklets, released under CC-BY-4.0)
- SA-1B (Segment Anything image dataset; supported alongside SA-V and DAVIS-style datasets like MOSE in training/README.md)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| facebook/sam2.1-hiera-tiny (38.9M, Apache-2.0) | safetensors | 0.156 GB | open | de431c404385 |
| facebook/sam2.1-hiera-small (46M, Apache-2.0) | safetensors | 0.184 GB | open | ee5bba1d82bb |
| facebook/sam2.1-hiera-base-plus (80.8M, Apache-2.0) | safetensors | 0.323 GB | open | b7320756a133 |
| facebook/sam2.1-hiera-large (224.4M, Apache-2.0) | safetensors | 0.898 GB | open | 665f8e2ad61c |
All four improved SAM 2.1 checkpoints verified via Hugging Face API (each provides both model.safetensors and .pt under Apache-2.0, plus direct dl.fbaipublicfiles.com links). The four earlier July 2024 SAM 2.0 checkpoints (facebook/sam2-hiera-{tiny,small,base-plus,large}) are also published under Apache-2.0.
Is this for you?
- Use cases
- Promptable static-image segmentation (clicks, bounding boxes, masks) and automatic mask generation
- Multi-object video object segmentation (VOS) and masklet propagation with per-object independent tracking
- Downstream integration with open-vocabulary detectors (such as Grounding DINO or Florence-2) for robotic perception
- Fine-tuning on custom image/video datasets (e.g., MOSE or SA-V) using the Hydra-based training suite
- Known limitations
- Class-agnostic visual prompt model: requires spatial prompts (points, boxes, or masks) rather than text prompts unless paired with an external detector or VLM.
- Training/fine-tuning guide assumes 80 GB A100 GPUs (8-GPU single-node or SLURM multi-node setup) and requires extracting SA-V videos to JPEG frames.
- Installation compiles an optional custom CUDA kernel (cc_torch connected-component post-processing); while SAM 2 runs if compilation fails, some post-processing functionality is limited.
- Open Source Robotics has not installed SAM 2, downloaded weights, run inference or fine-tuning, or reproduced SA-V/MOSE/LVOS benchmarks.
- Safety and limitations
- Perception-only segmentation model outputting 2D binary masks and video masklets; provides no 3D metric geometry or robot control safety guarantees.
- Masklet tracking can drift or drop objects under heavy occlusion or rapid appearance change without re-prompting.
- No independent latency, VRAM, or SA-V/MOSE/LVOS accuracy verification was performed by Open Source Robotics.
Compute & fine-tuning
- Inference compute
- Documented hardware: NVIDIA A100 (upstream benchmarks with torch 2.5.1, CUDA 12.4, bfloat16 autocast, torch.compile: 91.2 FPS Tiny / 84.8 FPS Small / 64.1 FPS Base+ / 39.5 FPS Large)
- Precision:
bf16 - Edge / consumer capable: Yes
Compact 38.9M (0.145 GB) Tiny variant is lightweight enough for edge GPU deployment, though upstream FPS numbers are measured on A100 and exact minimum VRAM is not stated.
- Fine-tuning
- Supported: Yes
- Methods: full training and fine-tuning on images, videos, or mixed datasets via Hydra training/train.py (single-node multi-GPU or SLURM multi-node)
- Minimum VRAM: 80 GB
- Recipe: Upstream fine-tuning documentation
training/README.md explicitly states 'We assume training on A100 GPUs with 80 GB of memory' and documents an 8-GPU MOSE fine-tuning example (expected Base+ J&F: 79.4).
- Host computer
CUDA-capable NVIDIA GPU (A100 used for upstream 39.5–91.2 FPS benchmarks; 0.156–0.898 GB safetensors checkpoint files); 80 GB A100 GPUs assumed for training/fine-tuning.
- Operating system
- Linux
- Windows (WSL with Ubuntu strongly recommended)
README recommends WSL with Ubuntu on Windows.
- Dependencies
- Python >=3.10, PyTorch >=2.5.1, TorchVision >=0.20.1, CUDA toolkit (nvcc)
- huggingface_hub (for from_pretrained loading)
- Optional: jupyter, matplotlib (notebooks); Hydra, tensorboard, SLURM (training); Docker (web demo)
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [1]
- Create a Python >=3.10 environment with PyTorch >=2.5.1, TorchVision >=0.20.1, and matching CUDA toolkit (nvcc) for optional custom CUDA post-processing kernel compilation.
- Clone facebookresearch/sam2 and install via pip install -e . (or pip install -e '.[notebooks]' / '.[dev]' for notebooks and training).
- Download SAM 2.1 checkpoints via checkpoints/download_ckpts.sh or load directly from Hugging Face via SAM2ImagePredictor.from_pretrained / SAM2VideoPredictor.from_pretrained.
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| SA-V test (J&F) | SAM 2.1: Tiny 76.5, Small 76.6, Base+ 78.2, Large 79.5 (vs SAM 2.0: 75.0 / 74.9 / 74.7 / 76.0). | Upstream-reported in README Model Description table; OSR has not run inference or reproduced SA-V metrics. |
| MOSE val (J&F) | SAM 2.1: Tiny 71.8, Small 73.5, Base+ 73.7, Large 74.6 (Base+ reaches 79.4 after MOSE fine-tuning per training/README.md). | Upstream-reported in README and training/README.md; OSR has not reproduced MOSE metrics. |
| LVOS v2 (J&F) | SAM 2.1: Tiny 77.3, Small 78.3, Base+ 78.2, Large 80.6 (vs SAM 2.0: 75.3 / 76.4 / 75.8 / 79.8). | Upstream-reported in README Model Description table; OSR has not reproduced LVOS v2 metrics. |
Project notes
Architecture, checkpoints, and workflows
SAM 2 pairs a hierarchical Vision Transformer (Hiera) image encoder with a prompt encoder, mask decoder, and streaming memory attention bank, treating images as single-frame videos. Users prompt SAM2ImagePredictor or SAM2VideoPredictor with points, bounding boxes, or masks in bfloat16 autocast to generate 2D segmentation masks and propagate multi-object masklets across video frames. Because SAM 2 is class-agnostic and takes spatial rather than text prompts, teams seeking language-conditioned object tracking must pair it with a separate open-vocabulary detector or VLM (editorial note). [1] [5]
Four ungated SAM 2.1 checkpoints (released September 2024) are verified on Hugging Face (each providing both model.safetensors and .pt, mirrored on Meta's CDN): facebook/sam2.1-hiera-tiny (38.9M, 0.156 GB), facebook/sam2.1-hiera-small (46M, 0.184 GB), facebook/sam2.1-hiera-base-plus (80.8M, 0.323 GB), and facebook/sam2.1-hiera-large (224.4M, 0.898 GB). The four earlier July 2024 SAM 2.0 checkpoints remain available as well. For customization, training/README.md provides a Hydra training pipeline (training/train.py) supporting mixed image/video training (SA-1B, SA-V, or DAVIS-style datasets like MOSE), assuming 80 GB A100 GPUs. [1] [4] [6]
Upstream-reported benchmarks and checkpoint tradeoffs
On an NVIDIA A100 with torch 2.5.1, CUDA 12.4, and full-model compilation, upstream reports SAM 2.1 speeds and J&F accuracy of Tiny: 91.2 FPS (76.5 SA-V test, 71.8 MOSE val, 77.3 LVOS v2); Small: 84.8 FPS (76.6 / 73.5 / 78.3); Base+: 64.1 FPS (78.2 / 73.7 / 78.2, reaching 79.4 on MOSE after fine-tuning); and Large: 39.5 FPS (79.5 / 74.6 / 80.6). Scoped to these upstream A100 measurements, Tiny delivers the highest throughput, Small raises all three J&F scores with a modest FPS drop, Base+ improves SA-V and MOSE over Small while slightly trailing Small on LVOS v2, and Large leads all three accuracy benchmarks at the lowest frame rate. The paper abstract also reports 3× fewer interactions in video segmentation than prior approaches and 6× faster image segmentation than SAM. [1] [5] [6]
Minimum inference VRAM is not established in the supplied evidence, and these A100 figures do not establish consumer-GPU or NVIDIA Jetson throughput (checkpoint file sizes on disk do not establish runtime GPU memory requirements). All figures are upstream-reported; Open Source Robotics has not installed SAM 2, downloaded weights, run training, fine-tuning, or inference, or reproduced any benchmarks. [1] [4]
What we checked
Source researched only; no installation, training, fine-tuning, inference, benchmark, or physical hardware test performed. No affiliate links or reused media.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Upstream last updated
- Dec 16, 2024 [3]
- Latest release
- Not established
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- commit snapshot; no GitHub release / 2b90b9f5ceec907a1c18123530e92e794ad901a4 · software repository
Separate license matrix
Include SAM 2 / SAM 2.1 as a permissively licensed (Apache-2.0) foundation segmentation model with open weights, full training/fine-tuning code, and CC-BY-4.0 SA-V video dataset.
| Layer | Availability / license |
|---|---|
| Code | available · Apache-2.0 (with optional cc_torch post-processing and SIL OFL 1.1 demo fonts) Inference, training, and demo code are Apache-2.0; optional cc_torch CUDA post-processing has LICENSE_cctorch and web demo fonts use SIL OFL 1.1. [1] [2] [3] |
| Model weights | available · Apache-2.0 All SAM 2.1 and SAM 2.0 checkpoints are ungated on Hugging Face and Meta CDN under Apache-2.0. [1] [2] [4] |
| Training datasets | available · CC-BY-4.0 (SA-V dataset) SA-V dataset (51K videos, 643K masklets) is released under CC-BY-4.0 via Meta's download portal. [7] |
| Documentation | available · Apache-2.0 README, INSTALL.md, training/README.md, sav_dataset/README.md, and Jupyter/Colab notebooks included. [1] [2] |
Permissive licensing and SA-V dataset
SAM 2's inference code, training code, web demo code, and all SAM 2 / SAM 2.1 model checkpoints are licensed under Apache-2.0, enabling commercial robotics integration without non-commercial restrictions (noting optional cc_torch post-processing under LICENSE_cctorch and web demo fonts under SIL OFL 1.1). The accompanying SA-V dataset (50,583 train, 155 val, and 150 test videos with 643K masklets) is released under CC-BY-4.0. [1] [2] [4] [7]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 SAM 2 README
SAM 2.1 and SAM 2.0 checkpoint tables (38.9M–224.4M, A100 FPS, SA-V/MOSE/LVOS J&F), torch.compile VOS optimization, installation, Hugging Face API, and Apache-2.0 licensing.Revision, date & where cited
AI at Meta, FAIR · github.com · revision
2b90b9f· Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 6 more - S002 SAM 2 LICENSE (Apache-2.0)
Apache License Version 2.0 covering SAM 2 checkpoints, demo code, and training code.Revision, date & where cited
Meta Platforms, Inc. · github.com · revision
2b90b9f· Accessed Nov 21, 2026
Used for: Separate license matrix - S003 GitHub repository metadata and pinned commit
Last commit 2024-12-16; no GitHub releases; Apache-2.0; not archived.Revision, date & where cited
facebookresearch/sam2 / GitHub · github.com · revision
2b90b9f· main / · Accessed Nov 21, 2026
Used for: Overview, What we checked, Separate license matrix - S004 Hugging Face SAM 2.1 checkpoint metadata (Tiny, Small, Base+, Large)
Verified all four ungated Apache-2.0 SAM 2.1 safetensors/.pt checkpoints: tiny (0.145 GB), small (0.172 GB), base-plus (0.301 GB), large (0.836 GB).Revision, date & where cited
AI at Meta / Hugging Face · huggingface.co · API metadata 2026-11-21 · Accessed Nov 21, 2026
Used for: Checkpoints, Compute & fine-tuning, Architecture, checkpoints, and workflows, Upstream-reported benchmarks and checkpoint tradeoffs and 1 more - S005 SAM 2: Segment Anything in Images and Videos (arXiv:2408.00714)
Streaming memory transformer architecture for promptable image and video segmentation, reporting 3x fewer interactions in video and 6x faster image segmentation than SAM.Revision, date & where cited
AI at Meta, FAIR (arXiv) · arxiv.org · arXiv:2408.00714v2 abstract · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Architecture, checkpoints, and workflows and 1 more - S006 SAM 2 Training Guide (training/README.md)
Hydra training/fine-tuning pipeline assuming 80 GB A100 GPUs, 8-GPU MOSE fine-tuning example (79.4 J&F), and mixed SA-1B/SA-V training setup.Revision, date & where cited
AI at Meta, FAIR · github.com · revision
2b90b9f· Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Compute & fine-tuning and 4 more - S007 SA-V Dataset README (sav_dataset/README.md)
SA-V dataset statistics (50,583 train / 155 val / 150 test videos, 643K masklets) released under CC-BY-4.0.Revision, date & where cited
AI at Meta, FAIR · github.com · revision
2b90b9f· Accessed Nov 21, 2026
Used for: Model card summary, Separate license matrix
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile for SAM 2 / SAM 2.1 (4 verified Apache-2.0 HF checkpoints, CC-BY-4.0 SA-V dataset).