Open Source RoboticsRobots, software & learning

Sparsh

Robotics model

Family of self-supervised Vision Transformer touch representations (MAE, DINO, DINOv2, I-JEPA, V-JEPA) pretrained on 460k+ vision-based tactile images for DIGIT and GelSight sensors. [1] [4]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Tactile sensing and robotic manipulation researchers working with DIGIT or GelSight sensors under non-commercial terms [1] [2]
Parameters
ViT-Small (~87.6 MB FP32 safetensors backbone) and ViT-Base (~345.0 MB FP32 safetensors backbone) [1] [5]
Embodiments
DIGIT vision-based tactile sensor (including multi-finger robot hand fingertip mounts), GelSight'17 (with markers)… [1] [4] [6]
Source rights
Named licenses are documented by part. Includes noncommercial terms. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware
License
CC-BY-NC-4.0 (noncommercial) · Details by part

Requirements · Research details and dates

On this page

Overview

Sparsh (facebookresearch/sparsh, CoRL 2024, arXiv:2410.24090) is a family of self-supervised Vision Transformer (ViT) touch representations developed by AI at Meta (FAIR), Carnegie Mellon University, and the University of Washington for vision-based tactile sensors including DIGIT, GelSight'17 (with markers), and GelSight Mini (without markers). At archived commit fee6a05a97330eca014b59c71ac41596fe3974a7, the repository provides PyTorch and Hydra pretraining code, TacBench frozen-encoder evaluation pipelines, a live force-field visualization demo, and links to 11 public Hugging Face checkpoints. [1] [3] [4] [5]

Model card summary

Architecture

Vision Transformer (ViT-Small and ViT-Base) tactile encoders pretrained via five self-supervised learning objectives (MAE, DINO, DINOv2, I-JEPA, V-JEPA). Each model takes a channel-concatenated pair of tactile images separated by a temporal stride of 5 frames (6 channels total, ~80 ms window at 60 FPS) with background subtraction, paired with lightweight task-specific decoders for downstream TacBench tasks.

[1] [4] [6]

Parameter count

ViT-Small (~87.6 MB FP32 safetensors backbone) and ViT-Base (~345.0 MB FP32 safetensors backbone); exact scalar parameter counts are not stated in README or model cards

Inferred from Hugging Face safetensors file sizes (~87.59 MB for ViT-Small and ~345.04 MB for ViT-Base across all 9 SSL backbones); exact parameter counts are not stated in README.

[1] [5] · Editorial assessment

Input modalities
tactile

Pairs of RGB vision-based tactile images (I_t concatenated with I_{t-5}, 6 channels) from DIGIT, GelSight'17, or GelSight Mini.

[1] [6]

Output modalities
6d-pose

Backbone checkpoints output dense ViT tactile feature embeddings (and released downstream decoders output normal/shear force fields) rather than robot control commands; mapped to 6d-pose as the closest controlled-vocabulary modality covered by TacBench T3 SE(3) pose estimation.

[1] [6] · Editorial assessment

Action space

Representation backbone and perception decoder (no direct robot action output); frozen encoders feed task-specific decoders for 3-axis normal/shear force fields, slip detection, object pose estimation, grasp stability, and textile classification.

[1] [4] [6]

Trained / supported embodiments
  • DIGIT vision-based tactile sensor (including multi-finger robot hand fingertip mounts)
  • GelSight'17 (with markers) and GelSight Mini (without markers) vision-based tactile sensors

[1] [4] [6]

Training datasets
  • 460k+ unlabeled vision-based tactile images combining YCB-Slide and Meta Touch-Slide (~360k DIGIT samples), Touch and Go (GelSight'17), and ObjectFolder-Real (GelSight Mini)
  • Downstream TacBench datasets: Meta DIGIT/GelSight Mini force & slip datasets, Meta DIGIT pose estimation dataset, Feeling of Success (~9k GelSight'17 grasp trials), and MIT Clothing dataset (4,467 GelSight'17 clips)

[1] [4] [7]

Interfaces
  • Python Hydra CLIs (train.py, train_task.py, test_task.py, demo_forcefield.py)
  • Hugging Face model & dataset hub repositories

[1] [5]

Checkpoints

CheckpointFormatSizeAccessPinned revision
facebook/sparsh-dino-basesafetensors1.977 GBopen89e38eaade8e
facebook/sparsh-dino-smallsafetensors0.766 GBopen1aa732e94c1d
facebook/sparsh-dinov2-basesafetensors1.977 GBopend682d6ed7d27
facebook/sparsh-ijepa-basesafetensors2.687 GBopen4f3316917b4e
facebook/sparsh-ijepa-smallsafetensors0.464 GBopen0b7242427535
facebook/sparsh-mae-basesafetensors1.583 GBopen00f6ba33797d
facebook/sparsh-mae-smallsafetensors0.621 GBopene8ca7c330133
facebook/sparsh-vjepa-basesafetensors3.119 GBopen8d5b22160577
facebook/sparsh-vjepa-smallsafetensors1.187 GBopen31ae99020102
facebook/sparsh-digit-forcefield-decoderpytorch0.546 GBopend7beed51ba30
facebook/sparsh-gelsight-forcefield-decoderpytorch0.546 GBopenebd4d0b09f1c

Each SSL backbone repo includes both a .safetensors backbone file and a full PyTorch Lightning .ckpt file; size_gb reflects total weight files in each HF repo as reported by the Hugging Face API.

[1] [5]

Is this for you?

Use cases
  • Frozen-encoder tactile feature extraction for force field estimation, slip detection, and SE(3) object pose estimation (TacBench)
  • Live normal and shear force-field visualization using a DIGIT or GelSight Mini sensor with released Sparsh (DINO) decoder checkpoints
  • Self-supervised pretraining on custom unlabeled vision-based tactile datasets using MAE, DINO, DINOv2, I-JEPA, or V-JEPA

[1] [4]

Known limitations
  • Code, model weights, and Meta-released datasets are licensed under CC-BY-NC-4.0, prohibiting commercial use.
  • The GitHub repository (facebookresearch/sparsh) is archived and no longer receives updates.
  • Pretraining experiments used 8x NVIDIA A100-80GB GPUs; downstream tasks require training or downloading task-specific decoders on top of the frozen encoder.
  • Requires background subtraction reference (no-contact) images for DIGIT preprocessing to handle sensor lighting and shadow distractors.

[1] [2] [3] [6]

Safety and limitations
  • CC-BY-NC-4.0 license restricts all code, weights, and Meta tactile datasets to non-commercial use.
  • Sparsh outputs tactile feature embeddings (and force-field maps when paired with a decoder) rather than direct robot arm or hand motion commands; downstream control and safety limits are external.
  • Repository is archived on GitHub as of review, so dependency drift or sensor driver incompatibilities will not be patched upstream.
  • Open Source Robotics has not installed dependencies, downloaded checkpoints, run training, fine-tuning, or inference, or tested Sparsh on physical tactile sensors.

[1] [2] [3] [6]

Compute & fine-tuning

Inference compute
  • Documented hardware: Not stated for inference (live demo runs on a single PC with DIGIT or GelSight Mini; all SSL pretraining experiments used 8x NVIDIA A100-80GB GPUs)

README specifies 8x A100-80GB for pretraining and a single PC for the live force-field demo, without stating a minimum inference VRAM or latency.

[1]

Fine-tuning
  • Supported: Yes
  • Methods: frozen-encoder downstream decoder training via train_task.py; full SSL pretraining via train.py (MAE, DINO, DINOv2, I-JEPA, V-JEPA)
  • Recipe: Upstream fine-tuning documentation

[1]

Host computer

Pretraining used 8x NVIDIA A100-80GB GPUs (batch size adjustable per GPU); live force-field demo runs on a PC connected to a DIGIT or GelSight Mini sensor.

[1]

Operating system
  • Linux

README references Linux webcam tools (Cheese, /dev/video*), bash dataset scripts, and SLURM submit scripts.

[1] · Editorial assessment

Dependencies
  • PyTorch and Mamba/Conda (environment.yml)
  • Hydra configuration framework and Weights & Biases (wandb)
  • digit-interface (for live DIGIT sensor demo)

[1]

Skills
  • PyTorch, Hydra configuration management, and Conda/Mamba environments
  • Vision-based tactile sensor interfacing (DIGIT digit-interface or Linux V4L2 webcam setup for GelSight Mini)

Inferred from README installation, Hydra config structure, and live demo instructions.

[1] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1] · Model card / documentation [1]

  • Clone facebookresearch/sparsh and create the tactile_ssl environment via mamba env create -f environment.yml.
  • Download pretrained SSL backbone and/or force-field decoder checkpoints from the Hugging Face facebook/sparsh collection.
  • Configure Hydra path and Weights & Biases YAML files under config/paths/ and config/wandb/.
  • Run train_task.py / test_task.py for TacBench tasks or demo_forcefield.py with a connected DIGIT (via digit-interface) or GelSight Mini webcam device.

[1]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
TacBench (6 vision-based tactile tasks across DIGIT, GelSight'17, and GelSight Mini; CoRL 2024, arXiv:2410.24090)SSL pre-training outperforms task- and sensor-specific end-to-end training by 95.1% on average across TacBench; Sparsh (DINO) and Sparsh (I-JEPA) are the most competitive variants.Upstream-reported aggregate improvement in arXiv:2410.24090 abstract using frozen encoders with task-specific decoders; OSR has not reproduced TacBench evaluations.

[1] [4]

Project notes

Architecture, checkpoints, and sensor setup

Sparsh adapts ViT-Small and ViT-Base encoders across five self-supervised learning objectives: MAE, DINO, I-JEPA, and V-JEPA each have both Small and Base releases, whereas DINOv2 is released in Base only (9 SSL backbones total). On the sparsh-dino-base model card, upstream describes the model as taking a channel-concatenated pair of tactile images separated by a temporal stride of 5 frames (6 channels total, representing an ~80 ms window at 60 FPS) with background subtraction to suppress lighting and shadow distractors. Preprocessing differs across sensors, however: the repository README notes that for Touch and Go GelSight'17 data, upstream performs no preprocessing because the sensor markers carry static shear information. Each of the 9 ungated SSL backbone repositories on Hugging Face ships a .safetensors backbone (roughly 87.6 MB for ViT-Small and 345.0 MB for ViT-Base) plus a full PyTorch Lightning .ckpt file, alongside 2 downstream force-field decoder checkpoints (sparsh-digit-forcefield-decoder and sparsh-gelsight-forcefield-decoder). [1] [5] [6]

Pretraining uses 460k+ unlabeled tactile images curated from YCB-Slide and Meta's Touch-Slide (~360k DIGIT samples), Touch and Go (GelSight'17), and ObjectFolder-Real (GelSight Mini), and upstream notes all pretraining runs used 8x NVIDIA A100-80GB GPUs. For downstream tasks, upstream primarily freezes the Sparsh encoder and trains lightweight task decoders via train_task.py. A live force-field demo (demo_forcefield.py) supports a single PC connected to a DIGIT sensor (via digit-interface) or a GelSight Mini webcam (/dev/video*), though exact minimum inference VRAM and latency are not stated. [1] [4] [7]

Upstream-reported TacBench performance

In arXiv:2410.24090, the authors introduce TacBench—six tactile tasks spanning force estimation, slip detection, SE(3) pose estimation, grasp stability, and textile recognition—and report that self-supervised Sparsh representations outperform task- and sensor-specific end-to-end training by 95.1% on average across TacBench, with Sparsh (DINO) and Sparsh (I-JEPA) performing best. All benchmark figures are upstream-reported; Open Source Robotics has not installed the environment, downloaded checkpoints, run training, fine-tuning, or inference, or tested Sparsh with physical tactile hardware. [1] [4]

What we checked

Source researched only; no installation, training, fine-tuning, inference, benchmark, or physical sensor test performed. No affiliate links or reused media.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Feb 27, 2025 [3]
Latest release
Not established
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
archived commit snapshot; no GitHub release / fee6a05a97330eca014b59c71ac41596fe3974a7 · software repository

Separate license matrix

Include Sparsh as a source-available, open-weights self-supervised tactile representation family with 11 verified Hugging Face checkpoints; clearly note its non-commercial CC-BY-NC-4.0 license and archived GitHub repository.

LayerAvailability / license
Codeavailable · CC-BY-NC-4.0
Archived repository licensed under Creative Commons Attribution-NonCommercial 4.0 International (LICENSE.md). [2] [3]
Model weightsavailable · CC-BY-NC-4.0
All 9 SSL ViT backbones and 2 force-field decoders are publicly downloadable on Hugging Face (ungated) under cc-by-nc-4.0. [1] [5]
Training datasetsavailable · CC-BY-NC-4.0
Meta's pretraining dataset (facebook/touch-slide) is verified public under cc-by-nc-4.0; separate license notices for Meta's three downstream task datasets (digit-force-estimation, gelsight-force-estimation, digit-pose-estimation) and third-party datasets (YCB-Slide, Touch and Go, ObjectFolder-Real, Feeling of Success, MIT Clothing) were not verified in this review. [1] [7]
Documentationavailable · CC-BY-NC-4.0
Repository README, Hydra configs, TacBench report notebook, and Hugging Face model cards. [1] [2]

Non-commercial license and archived status

The GitHub repository's LICENSE.md, all 11 Hugging Face model repositories, and the verified facebook/touch-slide pretraining dataset are licensed under Creative Commons Attribution-NonCommercial 4.0 International (CC-BY-NC-4.0), which prohibits commercial use. Separate license metadata for Meta's three downstream task datasets (digit-force-estimation, gelsight-force-estimation, and digit-pose-estimation) and third-party datasets linked in the README was not verified in this review. In addition, facebookresearch/sparsh is archived on GitHub (last commit 2025-02-27, no versioned releases), meaning users should expect no upstream bug fixes or dependency updates. [1] [2] [3] [5] [7]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile for Sparsh (11 Hugging Face checkpoints, CC-BY-NC-4.0, archived GitHub repo).