Grounded SAM 2
Robotics modelOpen-vocabulary detection, segmentation, and video object tracking pipeline from IDEA-Research combining open-set detectors (Grounding DINO 1.0, Florence-2, or cloud-API Grounding DINO 1.5/1.6 and DINO-X) with Meta's SAM 2 / SAM 2.1. [1] [5]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Robotics perception engineers building language-conditioned object detection, mask generation, and real-time camera tracking pipelines; Dataset builders running Florence-2 + SAM 2 cascaded auto-labeling workflows [1] [2]
- Embodiments
- Embodiment-agnostic 2D RGB open-vocabulary perception module (images, video files, and live camera streams) [1]
- Source rights
- Named licenses are documented by part. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
Overview
Grounded SAM 2 (IDEA-Research/Grounded-SAM-2, release v1.0, pinned commit b7a9c29f196edff0eb54dbe14588d7ae5e3dde28, last updated November 2025) is an open-world perception pipeline that chains text-prompted object detectors with Meta's SAM 2 / SAM 2.1 for image segmentation, RLE JSON export, auto-labeling, and video object tracking. Rather than introducing a newly trained unified checkpoint, it assembles open-world models into Python demos and a Docker workflow. [1] [3] [5]
Model card summary
- Architecture
Modular two-stage open-world perception pipeline chaining an open-vocabulary detector/VLM (local/HF Grounding DINO 1.0 Swin-T/Swin-B, Microsoft Florence-2, or cloud-API Grounding DINO 1.5/1.6 and DINO-X) to generate bounding boxes or phrase-grounded regions that prompt Meta's SAM 2 / SAM 2.1 image and video predictors for mask segmentation and multi-object tracking.
- Base model
SAM 2 / SAM 2.1 + Grounding DINO 1.0 (or Florence-2; optional cloud-API Grounding DINO 1.5/1.6 or DINO-X)
Assembles existing open-weight checkpoints rather than releasing a newly trained unified checkpoint.
- Parameter count
Composite pipeline: SAM 2.1 (38.9M–224.4M; 0.156–0.898 GB safetensors) + Grounding DINO 1.0 Tiny (0.689 GB) / Base (0.933 GB) or Florence-2-large (1.554 GB)
Exact combined parameter count depends on the selected detector + SAM 2.1 variant; exact HF safetensors sizes verified via API.
- Input modalities
- rgblanguage
RGB images, video files, or live camera streams paired with natural-language text prompts or Florence-2 task prompts.
- Output modalities
- video-frames
Outputs 2D bounding boxes, RLE-encoded segmentation masks (JSON), and tracked video masklets rather than robot control commands.
- Action space
Perception-only open-vocabulary segmentation and tracking pipeline (no robot action output); produces 2D bounding boxes, RLE masks, confidence scores, and tracked video object IDs.
- Trained / supported embodiments
- Embodiment-agnostic 2D RGB open-vocabulary perception module (images, video files, and live camera streams)
- Training datasets
- No joint Grounded SAM 2 training dataset; inherits upstream pre-training datasets of SAM 2 (SA-V, SA-1B), Grounding DINO 1.0, and Florence-2
Grounded SAM 2 assembles pretrained open-world models without additional joint training.
- Interfaces
- Python CLI demos for image segmentation, JSON RLE export, video tracking, continuous-ID live camera tracking, and Florence-2 auto-labeling
- Docker image workflow (Makefile)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| IDEA-Research/grounding-dino-tiny (open-set detector, Apache-2.0) | safetensors | 0.689 GB | open | a2bb814dd30d |
| IDEA-Research/grounding-dino-base (open-set detector, Apache-2.0) | safetensors | 0.933 GB | open | 12bdfa3120f3 |
| microsoft/Florence-2-large (vision-language grounder, MIT) | safetensors | 1.554 GB | open | 21a599d414c4 |
| facebook/sam2.1-hiera-large (segmentation/tracking backend, Apache-2.0) | safetensors | 0.898 GB | open | 665f8e2ad61c |
| facebook/sam2.1-hiera-tiny (lightweight segmentation backend, Apache-2.0) | safetensors | 0.156 GB | open | de431c404385 |
Verified open-weight Hugging Face checkpoints used by Grounded SAM 2 local/HF demos (gdino_checkpoints/download_ckpts.sh also fetches GroundingDINO v0.1.0-alpha .pth releases and checkpoints/download_ckpts.sh fetches all 4 SAM 2.1 .pt weights). Optional Grounding DINO 1.5/1.6 and DINO-X backends are closed cloud APIs via dds-cloudapi-sdk.
Is this for you?
- Use cases
- Text-prompted open-vocabulary image detection and RLE mask export (Grounding DINO 1.0 or Florence-2 + SAM 2.1)
- Video object tracking with box, point, or mask prompts, including continuous-ID tracking for newly appearing objects and live camera streams
- Cascaded image auto-labeling (Florence-2 captioning + phrase grounding + SAM 2 masks)
- Optional high-resolution tiled inference (SAHI) using cloud-API Grounding DINO 1.5 (supported on Grounding DINO 1.5 only), plus cloud-API Grounding DINO 1.6 and DINO-X demos
- Known limitations
- Assembly pipeline rather than a newly trained joint checkpoint: relies on separately trained Grounding DINO 1.0 / Florence-2 and SAM 2 / SAM 2.1 weights (no end-to-end joint training recipe in this repo).
- Stronger Grounding DINO 1.5/1.6, DINO-X, and SAHI demos require a proprietary Deep Data Space cloud API token (dds-cloudapi-sdk) rather than local open weights.
- Local Grounding DINO installation requires compiling CUDA Deformable Attention operators (CUDA_HOME matching PyTorch); README notes continuous-ID new-object tracking is still under development and not yet fully stable.
- Open Source Robotics has not installed Grounded SAM 2, downloaded weights, run inference or tracking demos, or benchmarked throughput.
- Safety and limitations
- Perception-only 2D open-vocabulary detection and segmentation pipeline; outputs 2D boxes and masks without 3D metric guarantees or robot safety bounds.
- Open-vocabulary detectors can hallucinate false positives or miss small/occluded objects depending on box_threshold and text_threshold tuning; continuous-ID tracking is explicitly marked under development.
- Using the optional Grounding DINO 1.5/1.6 or DINO-X backends sends images to an external cloud API (dds-cloudapi-sdk) with separate terms and latency.
- No independent accuracy, latency, or VRAM verification was performed by Open Source Robotics.
Compute & fine-tuning
- Inference compute
- Documented hardware: NVIDIA CUDA GPU strongly recommended (CUDA 12.1 + Deformable Attention CUDA build for local Grounding DINO; HF Grounding DINO demo avoids custom CUDA build)
- Precision:
fp32 / bf16 - Edge / consumer capable: No
Exact minimum VRAM and end-to-end latency are not specified in the README.
- Fine-tuning
- Supported: No
No joint Grounded SAM 2 fine-tuning recipe is documented; users fine-tune SAM 2 or Grounding DINO separately upstream.
- Host computer
NVIDIA CUDA GPU strongly recommended (combined open-weight checkpoints ~0.8–2.3 GB depending on detector and SAM 2.1 size); Docker container supported.
- Operating system
- Linux
CUDA Deformable Attention compilation and Docker Makefile target Linux.
[1] · Editorial assessment
- Dependencies
- Python 3.10, PyTorch >=2.3.1, TorchVision >=0.18.1, CUDA 12.1
- Hugging Face transformers (for HF Grounding DINO and Florence-2 demos) and Roboflow supervision
- Optional: dds-cloudapi-sdk (API V2 token required for Grounding DINO 1.5/1.6 and DINO-X); Docker
- Skills
- Python, PyTorch, and CUDA extension building (CUDA_HOME for Deformable Attention)
- Open-vocabulary detection thresholding and video tracking pipelines
Inferred from installation and tracking demo instructions.
[1] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [1]
- Install Python 3.10 with PyTorch >=2.3.1, TorchVision >=0.18.1, and CUDA 12.1 (or use the provided Makefile Docker workflow: make build-image && make run).
- Download SAM 2.1 weights (checkpoints/download_ckpts.sh) and optionally Grounding DINO 1.0 weights (gdino_checkpoints/download_ckpts.sh).
- Install SAM 2 (pip install -e .) and, for local Grounding DINO, set CUDA_HOME and run pip install --no-build-isolation -e grounding_dino.
- Run grounded_sam2_hf_model_demo.py, grounded_sam2_local_demo.py, grounded_sam2_florence2_image_demo.py, or grounded_sam2_tracking_demo.py.
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| Component zero-shot detection baseline (Grounding DINO 1.0, arXiv:2303.05499) | Grounding DINO 1.0 reports 52.5 AP zero-shot on COCO and 26.1 mean AP on ODinW (Grounded SAM 2 README does not publish a separate end-to-end numeric benchmark table). | Upstream-reported component score from Grounding DINO paper cited in README; OSR has not run Grounded SAM 2 or reproduced component metrics. |
Project notes
Architecture, open-weight vs. cloud backends, and workflows
Grounded SAM 2 operates in two stages: an open-vocabulary detector or vision-language model generates 2D bounding boxes or phrase-grounded regions from text prompts, and those detections prompt SAM 2 / SAM 2.1 (as box, sampled-point, or mask prompts) to produce pixel-level segmentation masks and track object masklets across video frames. The repository supports both standard first-frame video prompting and newer continuous-ID tracking scripts (including reverse tracking and live camera streams), though the README explicitly warns that continuous-ID detection of newly appearing objects is still under development and "not that stable now." [1]
Two deployment tiers exist with very different openness profiles: - Local open-weight tier: combines Hugging Face or local Grounding DINO 1.0 (IDEA-Research/grounding-dino-tiny 0.689 GB or grounding-dino-base 0.933 GB safetensors, Apache-2.0) or Florence-2 (microsoft/Florence-2-large 1.554 GB safetensors, MIT, also supporting caption-to-phrase-grounding auto-labeling) with SAM 2.1 (facebook/sam2.1-hiera-tiny 0.156 GB through large 0.898 GB safetensors, Apache-2.0). Local Grounding DINO requires compiling CUDA Deformable Attention (CUDA_HOME), whereas the Hugging Face transformers demo avoids custom CUDA builds. - Cloud-API tier: what the maintainers describe as stronger Grounding DINO 1.5 / 1.6 (with SAHI tiled inference supported only on Grounding DINO 1.5 for dense high-resolution images) and DINO-X backends are not open-weight; they require a Deep Data Space API token via dds-cloudapi-sdk (API V2). [1] [4] [6]
Minimum GPU VRAM and end-to-end latency are unspecified in the README and independently unmeasured (checkpoint file sizes on disk do not establish runtime GPU memory requirements), and cloud API pricing for dds-cloudapi-sdk is not established by this review. [1] [4]
Upstream-reported benchmarks
The Grounded SAM 2 repository does not publish a separate end-to-end benchmark table; it cites component papers including Grounding DINO 1.0 (arXiv:2303.05499, reporting 52.5 AP zero-shot on COCO and 26.1 mean AP on ODinW) and SAM 2 (arXiv:2408.00714). All figures are upstream-reported; Open Source Robotics has not installed Grounded SAM 2, downloaded weights, run training, fine-tuning, or inference, or reproduced component or pipeline benchmarks. [1] [5]
What we checked
Source researched only; no installation, training, fine-tuning, inference, benchmark, or physical hardware test performed. No affiliate links or reused media.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
Separate license matrix
Include Grounded SAM 2 as a widely used Apache-2.0 open-vocabulary segmentation and tracking pipeline with fully permissive local weights (Grounding DINO 1.0, Florence-2, SAM 2.1), marked partial because its stronger Grounding DINO 1.5/1.6 and DINO-X backends are closed cloud APIs.
| Layer | Availability / license |
|---|---|
| Code | available · Apache-2.0 (root LICENSE, LICENSE_sam2, LICENSE_groundingdino; plus LICENSE_cctorch) Repository code, vendored SAM 2, and vendored Grounding DINO 1.0 are Apache-2.0. [2] [3] |
| Model weights | available · Apache-2.0 (Grounding DINO 1.0 & SAM 2.1) / MIT (Florence-2); Grounding DINO 1.5/1.6 & DINO-X are closed cloud APIs Local open-weight path (Grounding DINO 1.0 + SAM 2.1 or Florence-2 + SAM 2.1) is fully permissive (Apache-2.0 / MIT), while Grounding DINO 1.5/1.6 and DINO-X weights are not released and require dds-cloudapi-sdk API tokens. [1] [2] [4] [6] |
| Documentation | available · Apache-2.0 README, INSTALL.md, and SAM2_README.md cover local, HF, Florence-2, Docker, and cloud-API demos. [1] [2] |
Assembly/inference pipeline with no newly created training dataset.
Licensing summary
The repository code (LICENSE), vendored SAM 2 (LICENSE_sam2), and vendored Grounding DINO 1.0 (LICENSE_groundingdino) carry Apache-2.0 licenses, and the named local component checkpoints carry Apache-2.0 (Grounding DINO 1.0 and SAM 2.1) or MIT (Florence-2) licenses; a whole-pipeline commercial clearance still requires reviewing remaining dependency terms (including optional cc_torch under LICENSE_cctorch). Adopting Grounding DINO 1.5/1.6 or DINO-X introduces a closed third-party cloud API dependency. [1] [2] [4]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 Grounded SAM 2 README
Installation, Docker workflow, local/HF Grounding DINO + SAM 2.1 demos, Florence-2 auto-labeling, live-camera continuous-ID tracking, and cloud-API Grounding DINO 1.5/1.6 & DINO-X integration.Revision, date & where cited
github.com · revision
b7a9c29· Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 5 more - S002 Grounded SAM 2 LICENSE (Apache-2.0)
Apache License Version 2.0 (alongside LICENSE_sam2 and LICENSE_groundingdino).Revision, date & where cited
github.com · revision
b7a9c29· Accessed Nov 21, 2026
Used for: Separate license matrix - S003 GitHub repository metadata, v1.0 release, and pinned commit
Last commit 2025-11-11; release v1.0 published 2024-12-11; Apache-2.0; not archived.Revision, date & where cited
IDEA-Research/Grounded-SAM-2 / GitHub · github.com · revision
b7a9c29· main / · Accessed Nov 21, 2026
Used for: Overview, What we checked, Separate license matrix - S004 Hugging Face component checkpoint metadata (Grounding DINO, Florence-2, SAM 2.1)
Verified ungated safetensors checkpoints: grounding-dino-tiny (0.642 GB, Apache-2.0), grounding-dino-base (0.869 GB, Apache-2.0), Florence-2-large (1.447 GB, MIT), sam2.1-hiera-large (0.836 GB, Apache-2.0), sam2.1-hiera-tiny (0.145 GB, Apache-2.0).Revision, date & where cited
IDEA-Research / Microsoft / Meta AI / Hugging Face · huggingface.co · API metadata 2026-11-21 · Accessed Nov 21, 2026
Used for: Model card summary, Checkpoints, Is this for you?, Compute & fine-tuning and 2 more - S005 Grounded SAM (arXiv:2401.14159) & Grounding DINO (arXiv:2303.05499)
Grounded SAM open-world assembly paradigm; Grounding DINO reports 52.5 AP zero-shot on COCO and 26.1 mean AP on ODinW.Revision, date & where cited
IDEA-Research (arXiv) · arxiv.org · arXiv:2401.14159v1 & arXiv:2303.05499v5 abstracts · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Upstream-reported benchmarks - S006 Grounded SAM 2 checkpoint download scripts
Scripts download Grounding DINO v0.1.0-alpha/alpha2 .pth releases and all four SAM 2.1 .pt checkpoints.Revision, date & where cited
github.com · revision
b7a9c29· gdino_checkpoints/download_ckpts.sh & checkpoints/download_ckpts.sh · Accessed Nov 21, 2026
Used for: Checkpoints, Setup & usage, Architecture, open-weight vs. cloud backends, and workflows, Separate license matrix
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile for Grounded SAM 2 (5 verified open-weight HF component checkpoints; partial openness due to cloud-API GD1.5/1.6 and DINO-X backends).