Sparsh
Robotics modelFamily of self-supervised Vision Transformer touch representations (MAE, DINO, DINOv2, I-JEPA, V-JEPA) pretrained on 460k+ vision-based tactile images for DIGIT and GelSight sensors. [1] [4]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Tactile sensing and robotic manipulation researchers working with DIGIT or GelSight sensors under non-commercial terms [1] [2]
- Parameters
- ViT-Small (~87.6 MB FP32 safetensors backbone) and ViT-Base (~345.0 MB FP32 safetensors backbone) [1] [5]
- Embodiments
- DIGIT vision-based tactile sensor (including multi-finger robot hand fingertip mounts), GelSight'17 (with markers)… [1] [4] [6]
- Source rights
- Named licenses are documented by part. Includes noncommercial terms. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- License
- CC-BY-NC-4.0 (noncommercial) · Details by part
Overview
Sparsh (facebookresearch/sparsh, CoRL 2024, arXiv:2410.24090) is a family of self-supervised Vision Transformer (ViT) touch representations developed by AI at Meta (FAIR), Carnegie Mellon University, and the University of Washington for vision-based tactile sensors including DIGIT, GelSight'17 (with markers), and GelSight Mini (without markers). At archived commit fee6a05a97330eca014b59c71ac41596fe3974a7, the repository provides PyTorch and Hydra pretraining code, TacBench frozen-encoder evaluation pipelines, a live force-field visualization demo, and links to 11 public Hugging Face checkpoints. [1] [3] [4] [5]
Model card summary
- Architecture
Vision Transformer (ViT-Small and ViT-Base) tactile encoders pretrained via five self-supervised learning objectives (MAE, DINO, DINOv2, I-JEPA, V-JEPA). Each model takes a channel-concatenated pair of tactile images separated by a temporal stride of 5 frames (6 channels total, ~80 ms window at 60 FPS) with background subtraction, paired with lightweight task-specific decoders for downstream TacBench tasks.
- Parameter count
ViT-Small (~87.6 MB FP32 safetensors backbone) and ViT-Base (~345.0 MB FP32 safetensors backbone); exact scalar parameter counts are not stated in README or model cards
Inferred from Hugging Face safetensors file sizes (~87.59 MB for ViT-Small and ~345.04 MB for ViT-Base across all 9 SSL backbones); exact parameter counts are not stated in README.
- Input modalities
- tactile
Pairs of RGB vision-based tactile images (I_t concatenated with I_{t-5}, 6 channels) from DIGIT, GelSight'17, or GelSight Mini.
- Output modalities
- 6d-pose
Backbone checkpoints output dense ViT tactile feature embeddings (and released downstream decoders output normal/shear force fields) rather than robot control commands; mapped to 6d-pose as the closest controlled-vocabulary modality covered by TacBench T3 SE(3) pose estimation.
- Action space
Representation backbone and perception decoder (no direct robot action output); frozen encoders feed task-specific decoders for 3-axis normal/shear force fields, slip detection, object pose estimation, grasp stability, and textile classification.
- Trained / supported embodiments
- DIGIT vision-based tactile sensor (including multi-finger robot hand fingertip mounts)
- GelSight'17 (with markers) and GelSight Mini (without markers) vision-based tactile sensors
- Training datasets
- 460k+ unlabeled vision-based tactile images combining YCB-Slide and Meta Touch-Slide (~360k DIGIT samples), Touch and Go (GelSight'17), and ObjectFolder-Real (GelSight Mini)
- Downstream TacBench datasets: Meta DIGIT/GelSight Mini force & slip datasets, Meta DIGIT pose estimation dataset, Feeling of Success (~9k GelSight'17 grasp trials), and MIT Clothing dataset (4,467 GelSight'17 clips)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| facebook/sparsh-dino-base | safetensors | 1.977 GB | open | 89e38eaade8e |
| facebook/sparsh-dino-small | safetensors | 0.766 GB | open | 1aa732e94c1d |
| facebook/sparsh-dinov2-base | safetensors | 1.977 GB | open | d682d6ed7d27 |
| facebook/sparsh-ijepa-base | safetensors | 2.687 GB | open | 4f3316917b4e |
| facebook/sparsh-ijepa-small | safetensors | 0.464 GB | open | 0b7242427535 |
| facebook/sparsh-mae-base | safetensors | 1.583 GB | open | 00f6ba33797d |
| facebook/sparsh-mae-small | safetensors | 0.621 GB | open | e8ca7c330133 |
| facebook/sparsh-vjepa-base | safetensors | 3.119 GB | open | 8d5b22160577 |
| facebook/sparsh-vjepa-small | safetensors | 1.187 GB | open | 31ae99020102 |
| facebook/sparsh-digit-forcefield-decoder | pytorch | 0.546 GB | open | d7beed51ba30 |
| facebook/sparsh-gelsight-forcefield-decoder | pytorch | 0.546 GB | open | ebd4d0b09f1c |
Each SSL backbone repo includes both a .safetensors backbone file and a full PyTorch Lightning .ckpt file; size_gb reflects total weight files in each HF repo as reported by the Hugging Face API.
Is this for you?
- Use cases
- Frozen-encoder tactile feature extraction for force field estimation, slip detection, and SE(3) object pose estimation (TacBench)
- Live normal and shear force-field visualization using a DIGIT or GelSight Mini sensor with released Sparsh (DINO) decoder checkpoints
- Self-supervised pretraining on custom unlabeled vision-based tactile datasets using MAE, DINO, DINOv2, I-JEPA, or V-JEPA
- Known limitations
- Code, model weights, and Meta-released datasets are licensed under CC-BY-NC-4.0, prohibiting commercial use.
- The GitHub repository (facebookresearch/sparsh) is archived and no longer receives updates.
- Pretraining experiments used 8x NVIDIA A100-80GB GPUs; downstream tasks require training or downloading task-specific decoders on top of the frozen encoder.
- Requires background subtraction reference (no-contact) images for DIGIT preprocessing to handle sensor lighting and shadow distractors.
- Safety and limitations
- CC-BY-NC-4.0 license restricts all code, weights, and Meta tactile datasets to non-commercial use.
- Sparsh outputs tactile feature embeddings (and force-field maps when paired with a decoder) rather than direct robot arm or hand motion commands; downstream control and safety limits are external.
- Repository is archived on GitHub as of review, so dependency drift or sensor driver incompatibilities will not be patched upstream.
- Open Source Robotics has not installed dependencies, downloaded checkpoints, run training, fine-tuning, or inference, or tested Sparsh on physical tactile sensors.
Compute & fine-tuning
- Inference compute
- Documented hardware: Not stated for inference (live demo runs on a single PC with DIGIT or GelSight Mini; all SSL pretraining experiments used 8x NVIDIA A100-80GB GPUs)
README specifies 8x A100-80GB for pretraining and a single PC for the live force-field demo, without stating a minimum inference VRAM or latency.
- Fine-tuning
- Supported: Yes
- Methods: frozen-encoder downstream decoder training via train_task.py; full SSL pretraining via train.py (MAE, DINO, DINOv2, I-JEPA, V-JEPA)
- Recipe: Upstream fine-tuning documentation
- Host computer
Pretraining used 8x NVIDIA A100-80GB GPUs (batch size adjustable per GPU); live force-field demo runs on a PC connected to a DIGIT or GelSight Mini sensor.
- Operating system
- Linux
README references Linux webcam tools (Cheese, /dev/video*), bash dataset scripts, and SLURM submit scripts.
[1] · Editorial assessment
- Dependencies
- PyTorch and Mamba/Conda (environment.yml)
- Hydra configuration framework and Weights & Biases (wandb)
- digit-interface (for live DIGIT sensor demo)
- Skills
- PyTorch, Hydra configuration management, and Conda/Mamba environments
- Vision-based tactile sensor interfacing (DIGIT digit-interface or Linux V4L2 webcam setup for GelSight Mini)
Inferred from README installation, Hydra config structure, and live demo instructions.
[1] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [1]
- Clone facebookresearch/sparsh and create the tactile_ssl environment via mamba env create -f environment.yml.
- Download pretrained SSL backbone and/or force-field decoder checkpoints from the Hugging Face facebook/sparsh collection.
- Configure Hydra path and Weights & Biases YAML files under config/paths/ and config/wandb/.
- Run train_task.py / test_task.py for TacBench tasks or demo_forcefield.py with a connected DIGIT (via digit-interface) or GelSight Mini webcam device.
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| TacBench (6 vision-based tactile tasks across DIGIT, GelSight'17, and GelSight Mini; CoRL 2024, arXiv:2410.24090) | SSL pre-training outperforms task- and sensor-specific end-to-end training by 95.1% on average across TacBench; Sparsh (DINO) and Sparsh (I-JEPA) are the most competitive variants. | Upstream-reported aggregate improvement in arXiv:2410.24090 abstract using frozen encoders with task-specific decoders; OSR has not reproduced TacBench evaluations. |
Project notes
Architecture, checkpoints, and sensor setup
Sparsh adapts ViT-Small and ViT-Base encoders across five self-supervised learning objectives: MAE, DINO, I-JEPA, and V-JEPA each have both Small and Base releases, whereas DINOv2 is released in Base only (9 SSL backbones total). On the sparsh-dino-base model card, upstream describes the model as taking a channel-concatenated pair of tactile images separated by a temporal stride of 5 frames (6 channels total, representing an ~80 ms window at 60 FPS) with background subtraction to suppress lighting and shadow distractors. Preprocessing differs across sensors, however: the repository README notes that for Touch and Go GelSight'17 data, upstream performs no preprocessing because the sensor markers carry static shear information. Each of the 9 ungated SSL backbone repositories on Hugging Face ships a .safetensors backbone (roughly 87.6 MB for ViT-Small and 345.0 MB for ViT-Base) plus a full PyTorch Lightning .ckpt file, alongside 2 downstream force-field decoder checkpoints (sparsh-digit-forcefield-decoder and sparsh-gelsight-forcefield-decoder). [1] [5] [6]
Pretraining uses 460k+ unlabeled tactile images curated from YCB-Slide and Meta's Touch-Slide (~360k DIGIT samples), Touch and Go (GelSight'17), and ObjectFolder-Real (GelSight Mini), and upstream notes all pretraining runs used 8x NVIDIA A100-80GB GPUs. For downstream tasks, upstream primarily freezes the Sparsh encoder and trains lightweight task decoders via train_task.py. A live force-field demo (demo_forcefield.py) supports a single PC connected to a DIGIT sensor (via digit-interface) or a GelSight Mini webcam (/dev/video*), though exact minimum inference VRAM and latency are not stated. [1] [4] [7]
Upstream-reported TacBench performance
In arXiv:2410.24090, the authors introduce TacBench—six tactile tasks spanning force estimation, slip detection, SE(3) pose estimation, grasp stability, and textile recognition—and report that self-supervised Sparsh representations outperform task- and sensor-specific end-to-end training by 95.1% on average across TacBench, with Sparsh (DINO) and Sparsh (I-JEPA) performing best. All benchmark figures are upstream-reported; Open Source Robotics has not installed the environment, downloaded checkpoints, run training, fine-tuning, or inference, or tested Sparsh with physical tactile hardware. [1] [4]
What we checked
Source researched only; no installation, training, fine-tuning, inference, benchmark, or physical sensor test performed. No affiliate links or reused media.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Upstream last updated
- Feb 27, 2025 [3]
- Latest release
- Not established
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- archived commit snapshot; no GitHub release / fee6a05a97330eca014b59c71ac41596fe3974a7 · software repository
Separate license matrix
Include Sparsh as a source-available, open-weights self-supervised tactile representation family with 11 verified Hugging Face checkpoints; clearly note its non-commercial CC-BY-NC-4.0 license and archived GitHub repository.
| Layer | Availability / license |
|---|---|
| Code | available · CC-BY-NC-4.0 Archived repository licensed under Creative Commons Attribution-NonCommercial 4.0 International (LICENSE.md). [2] [3] |
| Model weights | available · CC-BY-NC-4.0 All 9 SSL ViT backbones and 2 force-field decoders are publicly downloadable on Hugging Face (ungated) under cc-by-nc-4.0. [1] [5] |
| Training datasets | available · CC-BY-NC-4.0 Meta's pretraining dataset (facebook/touch-slide) is verified public under cc-by-nc-4.0; separate license notices for Meta's three downstream task datasets (digit-force-estimation, gelsight-force-estimation, digit-pose-estimation) and third-party datasets (YCB-Slide, Touch and Go, ObjectFolder-Real, Feeling of Success, MIT Clothing) were not verified in this review. [1] [7] |
| Documentation | available · CC-BY-NC-4.0 Repository README, Hydra configs, TacBench report notebook, and Hugging Face model cards. [1] [2] |
Non-commercial license and archived status
The GitHub repository's LICENSE.md, all 11 Hugging Face model repositories, and the verified facebook/touch-slide pretraining dataset are licensed under Creative Commons Attribution-NonCommercial 4.0 International (CC-BY-NC-4.0), which prohibits commercial use. Separate license metadata for Meta's three downstream task datasets (digit-force-estimation, gelsight-force-estimation, and digit-pose-estimation) and third-party datasets linked in the README was not verified in this review. In addition, facebookresearch/sparsh is archived on GitHub (last commit 2025-02-27, no versioned releases), meaning users should expect no upstream bug fixes or dependency updates. [1] [2] [3] [5] [7]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 Sparsh repository README
Installation via environment.yml, 9 SSL backbone HF links, 2 force-field decoder HF links, DIGIT/GelSight pretraining and TacBench datasets, 8x A100-80GB training hardware, and live force-field demo instructions.Revision, date & where cited
AI at Meta / facebookresearch · github.com · revision
fee6a05· Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 6 more - S002 Sparsh LICENSE.md
Full text of Creative Commons Attribution-NonCommercial 4.0 International Public License (CC-BY-NC-4.0).Revision, date & where cited
AI at Meta / facebookresearch · github.com · revision
fee6a05· Accessed Nov 21, 2026
Used for: Is this for you?, Separate license matrix - S003 GitHub repository metadata and pinned commit
Repository archived=true; last commit 2025-02-27; no GitHub releases.Revision, date & where cited
facebookresearch/sparsh / GitHub · github.com · revision
fee6a05· main / · Accessed Nov 21, 2026
Used for: Overview, Is this for you?, What we checked, Separate license matrix - S004 Sparsh: Self-supervised touch representations for vision-based tactile sensing (arXiv:2410.24090)
Pretraining on 460k+ tactile images with masking and self-distillation; reports 95.1% average improvement over end-to-end training across six TacBench tasks, with DINO and I-JEPA most competitive.Revision, date & where cited
AI at Meta, CMU, UW (CoRL 2024 / arXiv) · arxiv.org · arXiv:2410.24090v1 abstract · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Upstream-reported benchmarks and 2 more - S005 Hugging Face API metadata for all 11 Sparsh model checkpoints
Verified ungated access, cc-by-nc-4.0 license, exact commit SHAs, and safetensors/ckpt file sizes across all 9 SSL backbones and 2 force-field decoders.Revision, date & where cited
facebook / Hugging Face · huggingface.co · HF API blobs=true 2026-11-21 · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Architecture, checkpoints, and sensor setup and 1 more - S006 Sparsh DINO Base Hugging Face model card
Documents 6-channel paired tactile image input with temporal stride 5 (~80 ms window at 60 FPS), background subtraction preprocessing, and frozen-encoder decoder usage.Revision, date & where cited
facebook / Hugging Face · huggingface.co · README.md · Accessed Nov 21, 2026
Used for: Model card summary, Is this for you?, Architecture, checkpoints, and sensor setup - S007 Hugging Face dataset API metadata for facebook/touch-slide
Verified public ungated Touch-Slide DIGIT pretraining dataset under cc-by-nc-4.0.Revision, date & where cited
facebook / Hugging Face · huggingface.co · Accessed Nov 21, 2026
Used for: Model card summary, Architecture, checkpoints, and sensor setup, Separate license matrix
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile for Sparsh (11 Hugging Face checkpoints, CC-BY-NC-4.0, archived GitHub repo).