Open Source RoboticsRobots, software & learning

Grounded SAM 2

Robotics model

Open-vocabulary detection, segmentation, and video object tracking pipeline from IDEA-Research combining open-set detectors (Grounding DINO 1.0, Florence-2, or cloud-API Grounding DINO 1.5/1.6 and DINO-X) with Meta's SAM 2 / SAM 2.1. [1] [5]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Robotics perception engineers building language-conditioned object detection, mask generation, and real-time camera tracking pipelines; Dataset builders running Florence-2 + SAM 2 cascaded auto-labeling workflows [1] [2]
Parameters
Composite pipeline: SAM 2.1 (38.9M–224.4M [1] [4]
Embodiments
Embodiment-agnostic 2D RGB open-vocabulary perception module (images, video files, and live camera streams) [1]
Source rights
Named licenses are documented by part. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware

Requirements · Research details and dates

On this page

Overview

Grounded SAM 2 (IDEA-Research/Grounded-SAM-2, release v1.0, pinned commit b7a9c29f196edff0eb54dbe14588d7ae5e3dde28, last updated November 2025) is an open-world perception pipeline that chains text-prompted object detectors with Meta's SAM 2 / SAM 2.1 for image segmentation, RLE JSON export, auto-labeling, and video object tracking. Rather than introducing a newly trained unified checkpoint, it assembles open-world models into Python demos and a Docker workflow. [1] [3] [5]

Model card summary

Architecture

Modular two-stage open-world perception pipeline chaining an open-vocabulary detector/VLM (local/HF Grounding DINO 1.0 Swin-T/Swin-B, Microsoft Florence-2, or cloud-API Grounding DINO 1.5/1.6 and DINO-X) to generate bounding boxes or phrase-grounded regions that prompt Meta's SAM 2 / SAM 2.1 image and video predictors for mask segmentation and multi-object tracking.

[1] [5]

Base model

SAM 2 / SAM 2.1 + Grounding DINO 1.0 (or Florence-2; optional cloud-API Grounding DINO 1.5/1.6 or DINO-X)

Assembles existing open-weight checkpoints rather than releasing a newly trained unified checkpoint.

[1] [4]

Parameter count

Composite pipeline: SAM 2.1 (38.9M–224.4M; 0.156–0.898 GB safetensors) + Grounding DINO 1.0 Tiny (0.689 GB) / Base (0.933 GB) or Florence-2-large (1.554 GB)

Exact combined parameter count depends on the selected detector + SAM 2.1 variant; exact HF safetensors sizes verified via API.

[1] [4]

Input modalities
rgblanguage

RGB images, video files, or live camera streams paired with natural-language text prompts or Florence-2 task prompts.

[1]

Output modalities
video-frames

Outputs 2D bounding boxes, RLE-encoded segmentation masks (JSON), and tracked video masklets rather than robot control commands.

[1]

Action space

Perception-only open-vocabulary segmentation and tracking pipeline (no robot action output); produces 2D bounding boxes, RLE masks, confidence scores, and tracked video object IDs.

[1]

Trained / supported embodiments
  • Embodiment-agnostic 2D RGB open-vocabulary perception module (images, video files, and live camera streams)

[1]

Training datasets
  • No joint Grounded SAM 2 training dataset; inherits upstream pre-training datasets of SAM 2 (SA-V, SA-1B), Grounding DINO 1.0, and Florence-2

Grounded SAM 2 assembles pretrained open-world models without additional joint training.

[1] [5]

Interfaces
  • Python CLI demos for image segmentation, JSON RLE export, video tracking, continuous-ID live camera tracking, and Florence-2 auto-labeling
  • Docker image workflow (Makefile)

[1]

Checkpoints

Verified open-weight Hugging Face checkpoints used by Grounded SAM 2 local/HF demos (gdino_checkpoints/download_ckpts.sh also fetches GroundingDINO v0.1.0-alpha .pth releases and checkpoints/download_ckpts.sh fetches all 4 SAM 2.1 .pt weights). Optional Grounding DINO 1.5/1.6 and DINO-X backends are closed cloud APIs via dds-cloudapi-sdk.

[1] [4] [6]

Is this for you?

Use cases
  • Text-prompted open-vocabulary image detection and RLE mask export (Grounding DINO 1.0 or Florence-2 + SAM 2.1)
  • Video object tracking with box, point, or mask prompts, including continuous-ID tracking for newly appearing objects and live camera streams
  • Cascaded image auto-labeling (Florence-2 captioning + phrase grounding + SAM 2 masks)
  • Optional high-resolution tiled inference (SAHI) using cloud-API Grounding DINO 1.5 (supported on Grounding DINO 1.5 only), plus cloud-API Grounding DINO 1.6 and DINO-X demos

[1]

Known limitations
  • Assembly pipeline rather than a newly trained joint checkpoint: relies on separately trained Grounding DINO 1.0 / Florence-2 and SAM 2 / SAM 2.1 weights (no end-to-end joint training recipe in this repo).
  • Stronger Grounding DINO 1.5/1.6, DINO-X, and SAHI demos require a proprietary Deep Data Space cloud API token (dds-cloudapi-sdk) rather than local open weights.
  • Local Grounding DINO installation requires compiling CUDA Deformable Attention operators (CUDA_HOME matching PyTorch); README notes continuous-ID new-object tracking is still under development and not yet fully stable.
  • Open Source Robotics has not installed Grounded SAM 2, downloaded weights, run inference or tracking demos, or benchmarked throughput.

[1] [4]

Safety and limitations
  • Perception-only 2D open-vocabulary detection and segmentation pipeline; outputs 2D boxes and masks without 3D metric guarantees or robot safety bounds.
  • Open-vocabulary detectors can hallucinate false positives or miss small/occluded objects depending on box_threshold and text_threshold tuning; continuous-ID tracking is explicitly marked under development.
  • Using the optional Grounding DINO 1.5/1.6 or DINO-X backends sends images to an external cloud API (dds-cloudapi-sdk) with separate terms and latency.
  • No independent accuracy, latency, or VRAM verification was performed by Open Source Robotics.

[1]

Compute & fine-tuning

Inference compute
  • Documented hardware: NVIDIA CUDA GPU strongly recommended (CUDA 12.1 + Deformable Attention CUDA build for local Grounding DINO; HF Grounding DINO demo avoids custom CUDA build)
  • Precision: fp32 / bf16
  • Edge / consumer capable: No

Exact minimum VRAM and end-to-end latency are not specified in the README.

[1]

Fine-tuning
  • Supported: No

No joint Grounded SAM 2 fine-tuning recipe is documented; users fine-tune SAM 2 or Grounding DINO separately upstream.

[1]

Host computer

NVIDIA CUDA GPU strongly recommended (combined open-weight checkpoints ~0.8–2.3 GB depending on detector and SAM 2.1 size); Docker container supported.

[1] [4]

Operating system
  • Linux

CUDA Deformable Attention compilation and Docker Makefile target Linux.

[1] · Editorial assessment

Dependencies
  • Python 3.10, PyTorch >=2.3.1, TorchVision >=0.18.1, CUDA 12.1
  • Hugging Face transformers (for HF Grounding DINO and Florence-2 demos) and Roboflow supervision
  • Optional: dds-cloudapi-sdk (API V2 token required for Grounding DINO 1.5/1.6 and DINO-X); Docker

[1]

Skills
  • Python, PyTorch, and CUDA extension building (CUDA_HOME for Deformable Attention)
  • Open-vocabulary detection thresholding and video tracking pipelines

Inferred from installation and tracking demo instructions.

[1] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1] · Model card / documentation [1]

  • Install Python 3.10 with PyTorch >=2.3.1, TorchVision >=0.18.1, and CUDA 12.1 (or use the provided Makefile Docker workflow: make build-image && make run).
  • Download SAM 2.1 weights (checkpoints/download_ckpts.sh) and optionally Grounding DINO 1.0 weights (gdino_checkpoints/download_ckpts.sh).
  • Install SAM 2 (pip install -e .) and, for local Grounding DINO, set CUDA_HOME and run pip install --no-build-isolation -e grounding_dino.
  • Run grounded_sam2_hf_model_demo.py, grounded_sam2_local_demo.py, grounded_sam2_florence2_image_demo.py, or grounded_sam2_tracking_demo.py.

[1] [6]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
Component zero-shot detection baseline (Grounding DINO 1.0, arXiv:2303.05499)Grounding DINO 1.0 reports 52.5 AP zero-shot on COCO and 26.1 mean AP on ODinW (Grounded SAM 2 README does not publish a separate end-to-end numeric benchmark table).Upstream-reported component score from Grounding DINO paper cited in README; OSR has not run Grounded SAM 2 or reproduced component metrics.

[1] [5]

Project notes

Architecture, open-weight vs. cloud backends, and workflows

Grounded SAM 2 operates in two stages: an open-vocabulary detector or vision-language model generates 2D bounding boxes or phrase-grounded regions from text prompts, and those detections prompt SAM 2 / SAM 2.1 (as box, sampled-point, or mask prompts) to produce pixel-level segmentation masks and track object masklets across video frames. The repository supports both standard first-frame video prompting and newer continuous-ID tracking scripts (including reverse tracking and live camera streams), though the README explicitly warns that continuous-ID detection of newly appearing objects is still under development and "not that stable now." [1]

Two deployment tiers exist with very different openness profiles: - Local open-weight tier: combines Hugging Face or local Grounding DINO 1.0 (IDEA-Research/grounding-dino-tiny 0.689 GB or grounding-dino-base 0.933 GB safetensors, Apache-2.0) or Florence-2 (microsoft/Florence-2-large 1.554 GB safetensors, MIT, also supporting caption-to-phrase-grounding auto-labeling) with SAM 2.1 (facebook/sam2.1-hiera-tiny 0.156 GB through large 0.898 GB safetensors, Apache-2.0). Local Grounding DINO requires compiling CUDA Deformable Attention (CUDA_HOME), whereas the Hugging Face transformers demo avoids custom CUDA builds. - Cloud-API tier: what the maintainers describe as stronger Grounding DINO 1.5 / 1.6 (with SAHI tiled inference supported only on Grounding DINO 1.5 for dense high-resolution images) and DINO-X backends are not open-weight; they require a Deep Data Space API token via dds-cloudapi-sdk (API V2). [1] [4] [6]

Minimum GPU VRAM and end-to-end latency are unspecified in the README and independently unmeasured (checkpoint file sizes on disk do not establish runtime GPU memory requirements), and cloud API pricing for dds-cloudapi-sdk is not established by this review. [1] [4]

Upstream-reported benchmarks

The Grounded SAM 2 repository does not publish a separate end-to-end benchmark table; it cites component papers including Grounding DINO 1.0 (arXiv:2303.05499, reporting 52.5 AP zero-shot on COCO and 26.1 mean AP on ODinW) and SAM 2 (arXiv:2408.00714). All figures are upstream-reported; Open Source Robotics has not installed Grounded SAM 2, downloaded weights, run training, fine-tuning, or inference, or reproduced component or pipeline benchmarks. [1] [5]

What we checked

Source researched only; no installation, training, fine-tuning, inference, benchmark, or physical hardware test performed. No affiliate links or reused media.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Nov 11, 2025 [3]
Latest release
Dec 11, 2024 [3]
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
v1.0 (2024-12-11) / b7a9c29f196edff0eb54dbe14588d7ae5e3dde28 · software repository

Separate license matrix

Include Grounded SAM 2 as a widely used Apache-2.0 open-vocabulary segmentation and tracking pipeline with fully permissive local weights (Grounding DINO 1.0, Florence-2, SAM 2.1), marked partial because its stronger Grounding DINO 1.5/1.6 and DINO-X backends are closed cloud APIs.

LayerAvailability / license
Codeavailable · Apache-2.0 (root LICENSE, LICENSE_sam2, LICENSE_groundingdino; plus LICENSE_cctorch)
Repository code, vendored SAM 2, and vendored Grounding DINO 1.0 are Apache-2.0. [2] [3]
Model weightsavailable · Apache-2.0 (Grounding DINO 1.0 & SAM 2.1) / MIT (Florence-2); Grounding DINO 1.5/1.6 & DINO-X are closed cloud APIs
Local open-weight path (Grounding DINO 1.0 + SAM 2.1 or Florence-2 + SAM 2.1) is fully permissive (Apache-2.0 / MIT), while Grounding DINO 1.5/1.6 and DINO-X weights are not released and require dds-cloudapi-sdk API tokens. [1] [2] [4] [6]
Documentationavailable · Apache-2.0
README, INSTALL.md, and SAM2_README.md cover local, HF, Florence-2, Docker, and cloud-API demos. [1] [2]

Assembly/inference pipeline with no newly created training dataset.

Licensing summary

The repository code (LICENSE), vendored SAM 2 (LICENSE_sam2), and vendored Grounding DINO 1.0 (LICENSE_groundingdino) carry Apache-2.0 licenses, and the named local component checkpoints carry Apache-2.0 (Grounding DINO 1.0 and SAM 2.1) or MIT (Florence-2) licenses; a whole-pipeline commercial clearance still requires reviewing remaining dependency terms (including optional cc_torch under LICENSE_cctorch). Adopting Grounding DINO 1.5/1.6 or DINO-X introduces a closed third-party cloud API dependency. [1] [2] [4]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile for Grounded SAM 2 (5 verified open-weight HF component checkpoints; partial openness due to cloud-API GD1.5/1.6 and DINO-X backends).