Open Source RoboticsRobots, software & learning

Theia

Robotics model

Vision foundation backbone for robot learning that distills multiple off-the-shelf vision foundation models (CLIP, DINOv2, ViT, SAM, Depth Anything) into compact DeiT-based encoders. [1] [4] [5]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Robot learning researchers training visuomotor policies on pretrained visual encoders; Engineers evaluating multi-teacher visual distillation under non-commercial research terms [1] [2] [5]
Parameters
Not stated as exact scalars in README or arXiv:2407.20179 abstract [1] [4]
Embodiments
Embodiment-agnostic visual representation backbone (single-frame RGB input for downstream simulated or real robot… [1] [5]
Source rights
Named licenses are documented by part. Includes noncommercial terms. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware

Requirements · Research details and dates

On this page

Overview

Theia (bdaiinstitute/theia, CoRL 2024, arXiv:2407.20179) is a visual foundation backbone for robot learning that distills multiple off-the-shelf Vision Foundation Models (VFMs)—including CLIP, DINOv2, and ViT, plus SAM and Depth Anything in the cddsv variants—into compact DeiT patch16-224 encoders. At pinned commit a63701cdb17d6e1533d623ab72db6cdc788e5eaa, the repository provides Hugging Face Transformers loading instructions, multi-GPU distillation training scripts, and VFM feature decoding utilities. [1] [3] [4] [5]

Model card summary

Architecture

DeiT patch16-224 Vision Transformer student backbone (Tiny, Small, or Base) with translator heads trained to simultaneously distill intermediate spatial/semantic features from diverse Vision Foundation Model teachers (CLIP, DINOv2, ViT for cdiv; plus SAM and Depth Anything for cddsv).

[1] [4] [5]

Base model

facebook/deit-tiny-patch16-224, deit-small-patch16-224, and deit-base-patch16-224 backbones distilled from teacher VFMs (CLIP, DINOv2, ViT, SAM, Depth Anything)

[1] [4]

Parameter count

Not stated as exact scalars in README or arXiv:2407.20179 abstract; released in Tiny, Small, and Base DeiT patch16-224 variants (safetensors checkpoints range from 0.04 GB to 0.752 GB including translator heads)

Inferred from DeiT-Tiny/Small/Base backbone names and Hugging Face safetensors file sizes; weights were not loaded to count parameters.

[1] [4] · Editorial assessment

Input modalities
rgb

224x224x3 uint8 RGB image tensors.

[1]

Output modalities
video-frames

Theia is a visual representation backbone outputting intermediate feature tokens (forward_feature) and predicted teacher-VFM feature maps decodable to depth/segmentation/PCA visual outputs rather than direct robot actions; mapped to video-frames as the closest controlled-vocabulary visual modality.

[1] [4] · Editorial assessment

Action space

Visual foundation backbone only (no direct robot action output); produces intermediate visual representations (forward_feature) for downstream robot policy heads and a dictionary of predicted teacher-VFM feature tensors.

[1] [5]

Trained / supported embodiments
  • Embodiment-agnostic visual representation backbone (single-frame RGB input for downstream simulated or real robot policy heads)

[1] [5]

Training datasets
  • ImageNet (organized in webdataset format for distillation training)
  • Optional subsampled video frames and custom image datasets supported by preprocessing scripts; teacher feature normalization stats provided in feature_stats/

[1]

Interfaces
  • Hugging Face Transformers AutoModel API (trust_remote_code=True)
  • Python package (pip install -e .)
  • torchrun CLI training scripts
  • VFM decoding notebook and script

[1]

Checkpoints

CheckpointFormatSizeAccessPinned revision
theia-tiny-patch16-224-cdivsafetensors0.04 GBopen996d2eac80fb
theia-small-patch16-224-cdivsafetensors0.147 GBopenb07fb29f3fa0
theia-base-patch16-224-cdivsafetensors0.559 GBopen90eaf9cf0bf1
theia-tiny-patch16-224-cddsvsafetensors0.065 GBopen8ce99426c269
theia-small-patch16-224-cddsvsafetensors0.212 GBopen722a7af1d4c5
theia-base-patch16-224-cddsvsafetensors0.752 GBopenf251f815f620

Six public ungated Hugging Face safetensors checkpoints verified via Hugging Face API (cdiv = CLIP+DINOv2+ViT; cddsv = CLIP+DINOv2+DepthAnything+SAM+ViT).

[1] [4]

Is this for you?

Use cases
  • Frozen or fine-tuned visual feature extraction for robot manipulation and control policies via model.forward_feature()
  • Predicting teacher Vision Foundation Model features (CLIP, DINOv2, ViT, and optionally SAM and Depth Anything) from a single compact backbone
  • Decoding predicted VFM features into DINOv2 PCA visualizations, SAM segmentation masks, or Depth Anything depth maps (cddsv variants)

[1] [4] [5]

Known limitations
  • Repository LICENSE and Hugging Face model cards restrict use to non-commercial research purposes only.
  • Loading Hugging Face checkpoints via AutoModel.from_pretrained() requires trust_remote_code=True.
  • Distillation training requires preparing ImageNet (or custom image/video datasets) in webdataset format and extracting teacher VFM features.
  • Open Source Robotics has not installed, trained, fine-tuned, or run inference with Theia checkpoints.

[1] [2] [4]

Safety and limitations
  • The AI Institute License (Section 4) restricts software and weights to non-commercial research purposes only; for-profit enterprises may use it only under that non-commercial research limitation.
  • AutoModel.from_pretrained(..., trust_remote_code=True) executes remote repository Python code from Hugging Face; pin the commit revision and audit code before running.
  • Theia is a visual representation backbone only and does not output safe robot actions directly; downstream policy training and safety envelopes remain the user's responsibility.
  • No independent runtime, latency, or downstream robot policy benchmarks were executed by Open Source Robotics.

[1] [2] [4]

Compute & fine-tuning

Inference compute
  • Documented hardware: PyTorch / Hugging Face Transformers (CPU or CUDA GPU; README training example uses 8 GPUs via torchrun, while exact inference VRAM and latency are not specified)
  • Edge / consumer capable: No

README quick start runs on PyTorch without specifying minimum inference VRAM or latency; edge_capable left false because no edge benchmark is documented in README.

[1] · Editorial assessment

Fine-tuning
  • Supported: Yes
  • Methods: multi-teacher distillation training via torchrun train_rvfm.py; downstream robot policy training on extracted Theia features
  • Recipe: Upstream fine-tuning documentation

[1]

Host computer

Standard Python/PyTorch host for inference (compact 0.04–0.752 GB safetensors checkpoints); multi-GPU CUDA node for distillation training (README example uses torchrun --nproc_per_node=8).

Inferred from checkpoint sizes and README torchrun training command.

[1] [4] · Editorial assessment

Operating system
  • Linux

Dockerfile and Linux-oriented torchrun workflow documented; standard PyTorch/Transformers inference may run on other OSes.

[1] · Editorial assessment

Dependencies
  • Python >= 3.10
  • PyTorch and Hugging Face Transformers
  • Webdataset and safetensors (for distillation training)
  • Docker (optional)

[1]

Skills
  • Python and PyTorch
  • Hugging Face Transformers
  • Robot policy learning (for downstream control heads)

Inferred from README quick start and training workflow.

[1] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1]

  • Install Python >= 3.10 (or build the repository Dockerfile) and run pip install -e . for local development, or install torch and transformers for Hugging Face inference.
  • Load a checkpoint such as theaiinstitute/theia-base-patch16-224-cdiv via AutoModel.from_pretrained(..., trust_remote_code=True) pinned to a reviewed revision.
  • Call model.forward_feature(rgb_uint8_tensor) to extract robot-learning representations, or model(rgb_uint8_tensor) for predicted teacher-VFM features.
  • For distillation training, convert ImageNet or custom datasets to webdataset shards, extract teacher features, and run torchrun train_rvfm.py.

[1]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
CoRL 2024 Robot Learning & Visual Representation Benchmarks (arXiv:2407.20179)Upstream reports Theia (theia-<size>-patch16-224-cdiv) outperforms its teacher models and prior robot learning visual models using less training data and smaller model sizes.Upstream-reported paper abstract summary; exact per-benchmark numerical scores are not included in the supplied README or abstract, and OSR has not independently reproduced evaluations.

[1] [5]

Project notes

Architecture, checkpoints, and usage boundaries

Theia is a visual representation backbone rather than an end-to-end robot policy: it takes 224×224×3 uint8 RGB frames and exposes model.forward_feature() for downstream robot policy heads alongside model() to predict dictionaries of teacher-VFM feature tensors. Six public, ungated safetensors checkpoints are published on Hugging Face across Tiny, Small, and Base DeiT backbones in two teacher configurations: cdiv (CLIP, DINOv2, ViT; 0.04 GB, 0.147 GB, and 0.559 GB) and cddsv (adding SAM and Depth Anything; 0.065 GB, 0.212 GB, and 0.752 GB). [1] [4]

Loading the checkpoints through transformers.AutoModel.from_pretrained(..., trust_remote_code=True) executes custom repository code hosted on Hugging Face, so integrators should pin explicit commit revisions and audit the model code before execution. Exact scalar parameter counts and inference latency figures are not tabulated in the README or arXiv abstract. Training new distilled models requires Python 3.10+, preparing ImageNet or custom image/video datasets into webdataset shards, extracting teacher features, and running torchrun (train_rvfm.py; the README example uses 8 GPUs). [1] [4]

Upstream-reported evaluation

In the CoRL 2024 paper abstract (arXiv:2407.20179), the authors report that Theia (theia-<size>-patch16-224-cdiv) outperforms both its individual teacher models and prior robot-learning visual representations while using less training data and smaller model sizes. Exact per-benchmark numerical scores are not included in the supplied README or abstract. All evaluation claims are upstream-reported; Open Source Robotics has not installed dependencies, trained or fine-tuned models, run inference, or reproduced any robot-learning benchmarks. [1] [5]

What we checked

Source researched only; no installation, training, fine-tuning, inference, benchmark, or physical hardware test performed. No affiliate links or reused media.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Nov 6, 2025 [3]
Latest release
Not established
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
commit snapshot; no GitHub release / a63701cdb17d6e1533d623ab72db6cdc788e5eaa · software repository

Separate license matrix

Include the inspectable multi-teacher vision backbone and public safetensors weights under a non-commercial research scope; clearly distinguish RAI Institute's non-commercial research license from permissive open-source terms.

LayerAvailability / license
Codeavailable · The AI Institute License (custom BSD-style Non-Commercial Research License)
Section 4 restricts use to non-commercial research purposes only, including for-profit enterprises. [2] [3]
Model weightsavailable · The AI Institute License (Non-Commercial Research)
All 6 Hugging Face safetensors checkpoints are public and ungated, explicitly governed by The AI Institute non-commercial research license. [2] [4]
Training datasetsnot found · License not fully assessed
Pre-training uses external ImageNet (and optional user video datasets) that users must obtain separately under their own terms; repository provides preprocessing scripts and feature_stats/. [1]
Documentationavailable · The AI Institute License (Non-Commercial Research)
README, dataset format doc, and decoding example notebook included in repository. [1] [2]

Non-commercial research license

Although the repository LICENSE uses a BSD-style header, Section 4 explicitly restricts use to non-commercial research purposes only ("For profit enterprises may use the software, subject to this limitation"), and all six Hugging Face model cards reproduce the same AI Institute License for both the pre-trained model weights and code. External training datasets such as ImageNet are not bundled and retain their own separate terms. Teams needing a commercially permissive vision backbone cannot deploy Theia in production under its public license. [1] [2] [4]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile for Theia with 6 verified Hugging Face checkpoints and non-commercial license verification.