Open Source RoboticsRobots, software & learning

SAM 2 (Segment Anything Model 2 / 2.1)

Robotics model

Apache-2.0 foundation model from Meta FAIR for real-time promptable visual segmentation and multi-object tracking across static images and videos using a Hiera backbone and streaming memory transformer. [1] [2] [5]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Robotics perception engineers and researchers building object segmentation, instance tracking, or open-vocabulary perception pipelines (e.g., Grounded SAM 2); Commercial and academic teams seeking Apache-2.0 image/video segmentation weights and fine-tuning recipes [1] [2]
Parameters
38.9M (Hiera-Tiny), 46M (Hiera-Small), 80.8M (Hiera-Base+), 224.4M (Hiera-Large) [1]
Embodiments
Embodiment-agnostic 2D RGB image and video segmentation foundation model [1] [5]
Source rights
Named licenses are documented by part. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware

Requirements · Research details and dates

On this page

Overview

SAM 2 (facebookresearch/sam2, arXiv:2408.00714) is a foundation visual segmentation model from Meta FAIR that extends Segment Anything from static images to real-time multi-object video segmentation using a streaming memory transformer. At pinned commit 2b90b9f5ceec907a1c18123530e92e794ad901a4 (December 2024), the repository ships the improved SAM 2.1 checkpoints, full-model torch.compile VOS acceleration (vos_optimized=True), independent per-object tracking in SAM2VideoPredictor, Hydra-based training/fine-tuning code, and a deployable web demo. [1] [3] [5] [6]

Model card summary

Architecture

Hierarchical Vision Transformer (Hiera: Tiny 38.9M, Small 46M, Base+ 80.8M, Large 224.4M) image encoder coupled with a prompt encoder, mask decoder, and streaming memory attention bank enabling single-frame image segmentation and real-time multi-object video masklet propagation (with optional full-model torch.compile VOS optimization).

[1] [5]

Parameter count

38.9M (Hiera-Tiny), 46M (Hiera-Small), 80.8M (Hiera-Base+), 224.4M (Hiera-Large)

Exact parameter counts reported in README Model Description table for both SAM 2.1 and SAM 2 checkpoints.

[1]

Input modalities
rgb

Static RGB images or RGB video frames plus interactive spatial prompts (points, bounding boxes, or masks).

[1]

Output modalities
video-frames

Outputs per-frame binary segmentation masks and spatio-temporal multi-object video masklets rather than robot control commands.

[1]

Action space

Perception-only visual segmentation model (no robot action output); produces 2D segmentation masks for images and tracked multi-object masklets across video frames.

[1] [5]

Trained / supported embodiments
  • Embodiment-agnostic 2D RGB image and video segmentation foundation model

[1] [5]

Training datasets
  • SA-V (Segment Anything Video: 50,583 train / 155 val / 150 test videos, 643K masklets, released under CC-BY-4.0)
  • SA-1B (Segment Anything image dataset; supported alongside SA-V and DAVIS-style datasets like MOSE in training/README.md)

[1] [6] [7]

Interfaces
  • Python API (SAM2ImagePredictor, SAM2VideoPredictor, build_sam2, build_sam2_video_predictor)
  • Hugging Face Hub from_pretrained API
  • Hydra training/fine-tuning CLI (training/train.py) and SA-V evaluator (sav_evaluator.py)
  • Deployable frontend/backend web demo (Docker Compose)

[1] [6] [7]

Checkpoints

CheckpointFormatSizeAccessPinned revision
facebook/sam2.1-hiera-tiny (38.9M, Apache-2.0)safetensors0.156 GBopende431c404385
facebook/sam2.1-hiera-small (46M, Apache-2.0)safetensors0.184 GBopenee5bba1d82bb
facebook/sam2.1-hiera-base-plus (80.8M, Apache-2.0)safetensors0.323 GBopenb7320756a133
facebook/sam2.1-hiera-large (224.4M, Apache-2.0)safetensors0.898 GBopen665f8e2ad61c

All four improved SAM 2.1 checkpoints verified via Hugging Face API (each provides both model.safetensors and .pt under Apache-2.0, plus direct dl.fbaipublicfiles.com links). The four earlier July 2024 SAM 2.0 checkpoints (facebook/sam2-hiera-{tiny,small,base-plus,large}) are also published under Apache-2.0.

[1] [4]

Is this for you?

Use cases
  • Promptable static-image segmentation (clicks, bounding boxes, masks) and automatic mask generation
  • Multi-object video object segmentation (VOS) and masklet propagation with per-object independent tracking
  • Downstream integration with open-vocabulary detectors (such as Grounding DINO or Florence-2) for robotic perception
  • Fine-tuning on custom image/video datasets (e.g., MOSE or SA-V) using the Hydra-based training suite

[1] [6]

Known limitations
  • Class-agnostic visual prompt model: requires spatial prompts (points, boxes, or masks) rather than text prompts unless paired with an external detector or VLM.
  • Training/fine-tuning guide assumes 80 GB A100 GPUs (8-GPU single-node or SLURM multi-node setup) and requires extracting SA-V videos to JPEG frames.
  • Installation compiles an optional custom CUDA kernel (cc_torch connected-component post-processing); while SAM 2 runs if compilation fails, some post-processing functionality is limited.
  • Open Source Robotics has not installed SAM 2, downloaded weights, run inference or fine-tuning, or reproduced SA-V/MOSE/LVOS benchmarks.

[1] [6]

Safety and limitations
  • Perception-only segmentation model outputting 2D binary masks and video masklets; provides no 3D metric geometry or robot control safety guarantees.
  • Masklet tracking can drift or drop objects under heavy occlusion or rapid appearance change without re-prompting.
  • No independent latency, VRAM, or SA-V/MOSE/LVOS accuracy verification was performed by Open Source Robotics.

[1] [5]

Compute & fine-tuning

Inference compute
  • Documented hardware: NVIDIA A100 (upstream benchmarks with torch 2.5.1, CUDA 12.4, bfloat16 autocast, torch.compile: 91.2 FPS Tiny / 84.8 FPS Small / 64.1 FPS Base+ / 39.5 FPS Large)
  • Precision: bf16
  • Edge / consumer capable: Yes

Compact 38.9M (0.145 GB) Tiny variant is lightweight enough for edge GPU deployment, though upstream FPS numbers are measured on A100 and exact minimum VRAM is not stated.

[1] [4]

Fine-tuning
  • Supported: Yes
  • Methods: full training and fine-tuning on images, videos, or mixed datasets via Hydra training/train.py (single-node multi-GPU or SLURM multi-node)
  • Minimum VRAM: 80 GB
  • Recipe: Upstream fine-tuning documentation

training/README.md explicitly states 'We assume training on A100 GPUs with 80 GB of memory' and documents an 8-GPU MOSE fine-tuning example (expected Base+ J&F: 79.4).

[1] [6]

Host computer

CUDA-capable NVIDIA GPU (A100 used for upstream 39.5–91.2 FPS benchmarks; 0.156–0.898 GB safetensors checkpoint files); 80 GB A100 GPUs assumed for training/fine-tuning.

[1] [4] [6]

Operating system
  • Linux
  • Windows (WSL with Ubuntu strongly recommended)

README recommends WSL with Ubuntu on Windows.

[1]

Dependencies
  • Python >=3.10, PyTorch >=2.5.1, TorchVision >=0.20.1, CUDA toolkit (nvcc)
  • huggingface_hub (for from_pretrained loading)
  • Optional: jupyter, matplotlib (notebooks); Hydra, tensorboard, SLURM (training); Docker (web demo)

[1] [6]

Skills
  • Python and PyTorch (bfloat16 inference, torch.compile, CUDA toolkit setup)
  • Hydra configuration and multi-GPU/SLURM training for custom fine-tuning

Inferred from installation, inference, and training guides.

[1] [6] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1] · Model card / documentation [1]

  • Create a Python >=3.10 environment with PyTorch >=2.5.1, TorchVision >=0.20.1, and matching CUDA toolkit (nvcc) for optional custom CUDA post-processing kernel compilation.
  • Clone facebookresearch/sam2 and install via pip install -e . (or pip install -e '.[notebooks]' / '.[dev]' for notebooks and training).
  • Download SAM 2.1 checkpoints via checkpoints/download_ckpts.sh or load directly from Hugging Face via SAM2ImagePredictor.from_pretrained / SAM2VideoPredictor.from_pretrained.

[1] [6]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
SA-V test (J&F)SAM 2.1: Tiny 76.5, Small 76.6, Base+ 78.2, Large 79.5 (vs SAM 2.0: 75.0 / 74.9 / 74.7 / 76.0).Upstream-reported in README Model Description table; OSR has not run inference or reproduced SA-V metrics.
MOSE val (J&F)SAM 2.1: Tiny 71.8, Small 73.5, Base+ 73.7, Large 74.6 (Base+ reaches 79.4 after MOSE fine-tuning per training/README.md).Upstream-reported in README and training/README.md; OSR has not reproduced MOSE metrics.
LVOS v2 (J&F)SAM 2.1: Tiny 77.3, Small 78.3, Base+ 78.2, Large 80.6 (vs SAM 2.0: 75.3 / 76.4 / 75.8 / 79.8).Upstream-reported in README Model Description table; OSR has not reproduced LVOS v2 metrics.

[1] [6]

Project notes

Architecture, checkpoints, and workflows

SAM 2 pairs a hierarchical Vision Transformer (Hiera) image encoder with a prompt encoder, mask decoder, and streaming memory attention bank, treating images as single-frame videos. Users prompt SAM2ImagePredictor or SAM2VideoPredictor with points, bounding boxes, or masks in bfloat16 autocast to generate 2D segmentation masks and propagate multi-object masklets across video frames. Because SAM 2 is class-agnostic and takes spatial rather than text prompts, teams seeking language-conditioned object tracking must pair it with a separate open-vocabulary detector or VLM (editorial note). [1] [5]

Four ungated SAM 2.1 checkpoints (released September 2024) are verified on Hugging Face (each providing both model.safetensors and .pt, mirrored on Meta's CDN): facebook/sam2.1-hiera-tiny (38.9M, 0.156 GB), facebook/sam2.1-hiera-small (46M, 0.184 GB), facebook/sam2.1-hiera-base-plus (80.8M, 0.323 GB), and facebook/sam2.1-hiera-large (224.4M, 0.898 GB). The four earlier July 2024 SAM 2.0 checkpoints remain available as well. For customization, training/README.md provides a Hydra training pipeline (training/train.py) supporting mixed image/video training (SA-1B, SA-V, or DAVIS-style datasets like MOSE), assuming 80 GB A100 GPUs. [1] [4] [6]

Upstream-reported benchmarks and checkpoint tradeoffs

On an NVIDIA A100 with torch 2.5.1, CUDA 12.4, and full-model compilation, upstream reports SAM 2.1 speeds and J&F accuracy of Tiny: 91.2 FPS (76.5 SA-V test, 71.8 MOSE val, 77.3 LVOS v2); Small: 84.8 FPS (76.6 / 73.5 / 78.3); Base+: 64.1 FPS (78.2 / 73.7 / 78.2, reaching 79.4 on MOSE after fine-tuning); and Large: 39.5 FPS (79.5 / 74.6 / 80.6). Scoped to these upstream A100 measurements, Tiny delivers the highest throughput, Small raises all three J&F scores with a modest FPS drop, Base+ improves SA-V and MOSE over Small while slightly trailing Small on LVOS v2, and Large leads all three accuracy benchmarks at the lowest frame rate. The paper abstract also reports 3× fewer interactions in video segmentation than prior approaches and 6× faster image segmentation than SAM. [1] [5] [6]

Minimum inference VRAM is not established in the supplied evidence, and these A100 figures do not establish consumer-GPU or NVIDIA Jetson throughput (checkpoint file sizes on disk do not establish runtime GPU memory requirements). All figures are upstream-reported; Open Source Robotics has not installed SAM 2, downloaded weights, run training, fine-tuning, or inference, or reproduced any benchmarks. [1] [4]

What we checked

Source researched only; no installation, training, fine-tuning, inference, benchmark, or physical hardware test performed. No affiliate links or reused media.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Dec 16, 2024 [3]
Latest release
Not established
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
commit snapshot; no GitHub release / 2b90b9f5ceec907a1c18123530e92e794ad901a4 · software repository

Separate license matrix

Include SAM 2 / SAM 2.1 as a permissively licensed (Apache-2.0) foundation segmentation model with open weights, full training/fine-tuning code, and CC-BY-4.0 SA-V video dataset.

LayerAvailability / license
Codeavailable · Apache-2.0 (with optional cc_torch post-processing and SIL OFL 1.1 demo fonts)
Inference, training, and demo code are Apache-2.0; optional cc_torch CUDA post-processing has LICENSE_cctorch and web demo fonts use SIL OFL 1.1. [1] [2] [3]
Model weightsavailable · Apache-2.0
All SAM 2.1 and SAM 2.0 checkpoints are ungated on Hugging Face and Meta CDN under Apache-2.0. [1] [2] [4]
Training datasetsavailable · CC-BY-4.0 (SA-V dataset)
SA-V dataset (51K videos, 643K masklets) is released under CC-BY-4.0 via Meta's download portal. [7]
Documentationavailable · Apache-2.0
README, INSTALL.md, training/README.md, sav_dataset/README.md, and Jupyter/Colab notebooks included. [1] [2]

Permissive licensing and SA-V dataset

SAM 2's inference code, training code, web demo code, and all SAM 2 / SAM 2.1 model checkpoints are licensed under Apache-2.0, enabling commercial robotics integration without non-commercial restrictions (noting optional cc_torch post-processing under LICENSE_cctorch and web demo fonts under SIL OFL 1.1). The accompanying SA-V dataset (50,583 train, 155 val, and 150 test videos with 643K masklets) is released under CC-BY-4.0. [1] [2] [4] [7]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile for SAM 2 / SAM 2.1 (4 verified Apache-2.0 HF checkpoints, CC-BY-4.0 SA-V dataset).