Theia
Robotics modelVision foundation backbone for robot learning that distills multiple off-the-shelf vision foundation models (CLIP, DINOv2, ViT, SAM, Depth Anything) into compact DeiT-based encoders. [1] [4] [5]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Robot learning researchers training visuomotor policies on pretrained visual encoders; Engineers evaluating multi-teacher visual distillation under non-commercial research terms [1] [2] [5]
- Embodiments
- Embodiment-agnostic visual representation backbone (single-frame RGB input for downstream simulated or real robot… [1] [5]
- Source rights
- Named licenses are documented by part. Includes noncommercial terms. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- License
- The AI Institute License (custom BSD-style Non-Commercial Research License) (noncommercial), The AI Institute License (Non-Commercial Research) (noncommercial) · Details by part
Overview
Theia (bdaiinstitute/theia, CoRL 2024, arXiv:2407.20179) is a visual foundation backbone for robot learning that distills multiple off-the-shelf Vision Foundation Models (VFMs)—including CLIP, DINOv2, and ViT, plus SAM and Depth Anything in the cddsv variants—into compact DeiT patch16-224 encoders. At pinned commit a63701cdb17d6e1533d623ab72db6cdc788e5eaa, the repository provides Hugging Face Transformers loading instructions, multi-GPU distillation training scripts, and VFM feature decoding utilities. [1] [3] [4] [5]
Model card summary
- Architecture
DeiT patch16-224 Vision Transformer student backbone (Tiny, Small, or Base) with translator heads trained to simultaneously distill intermediate spatial/semantic features from diverse Vision Foundation Model teachers (CLIP, DINOv2, ViT for cdiv; plus SAM and Depth Anything for cddsv).
- Base model
facebook/deit-tiny-patch16-224, deit-small-patch16-224, and deit-base-patch16-224 backbones distilled from teacher VFMs (CLIP, DINOv2, ViT, SAM, Depth Anything)
- Parameter count
Not stated as exact scalars in README or arXiv:2407.20179 abstract; released in Tiny, Small, and Base DeiT patch16-224 variants (safetensors checkpoints range from 0.04 GB to 0.752 GB including translator heads)
Inferred from DeiT-Tiny/Small/Base backbone names and Hugging Face safetensors file sizes; weights were not loaded to count parameters.
- Input modalities
- rgb
224x224x3 uint8 RGB image tensors.
- Output modalities
- video-frames
Theia is a visual representation backbone outputting intermediate feature tokens (forward_feature) and predicted teacher-VFM feature maps decodable to depth/segmentation/PCA visual outputs rather than direct robot actions; mapped to video-frames as the closest controlled-vocabulary visual modality.
- Action space
Visual foundation backbone only (no direct robot action output); produces intermediate visual representations (forward_feature) for downstream robot policy heads and a dictionary of predicted teacher-VFM feature tensors.
- Trained / supported embodiments
- Embodiment-agnostic visual representation backbone (single-frame RGB input for downstream simulated or real robot policy heads)
- Training datasets
- ImageNet (organized in webdataset format for distillation training)
- Optional subsampled video frames and custom image datasets supported by preprocessing scripts; teacher feature normalization stats provided in feature_stats/
- Interfaces
- Hugging Face Transformers AutoModel API (trust_remote_code=True)
- Python package (pip install -e .)
- torchrun CLI training scripts
- VFM decoding notebook and script
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| theia-tiny-patch16-224-cdiv | safetensors | 0.04 GB | open | 996d2eac80fb |
| theia-small-patch16-224-cdiv | safetensors | 0.147 GB | open | b07fb29f3fa0 |
| theia-base-patch16-224-cdiv | safetensors | 0.559 GB | open | 90eaf9cf0bf1 |
| theia-tiny-patch16-224-cddsv | safetensors | 0.065 GB | open | 8ce99426c269 |
| theia-small-patch16-224-cddsv | safetensors | 0.212 GB | open | 722a7af1d4c5 |
| theia-base-patch16-224-cddsv | safetensors | 0.752 GB | open | f251f815f620 |
Six public ungated Hugging Face safetensors checkpoints verified via Hugging Face API (cdiv = CLIP+DINOv2+ViT; cddsv = CLIP+DINOv2+DepthAnything+SAM+ViT).
Is this for you?
- Use cases
- Frozen or fine-tuned visual feature extraction for robot manipulation and control policies via model.forward_feature()
- Predicting teacher Vision Foundation Model features (CLIP, DINOv2, ViT, and optionally SAM and Depth Anything) from a single compact backbone
- Decoding predicted VFM features into DINOv2 PCA visualizations, SAM segmentation masks, or Depth Anything depth maps (cddsv variants)
- Known limitations
- Repository LICENSE and Hugging Face model cards restrict use to non-commercial research purposes only.
- Loading Hugging Face checkpoints via AutoModel.from_pretrained() requires trust_remote_code=True.
- Distillation training requires preparing ImageNet (or custom image/video datasets) in webdataset format and extracting teacher VFM features.
- Open Source Robotics has not installed, trained, fine-tuned, or run inference with Theia checkpoints.
- Safety and limitations
- The AI Institute License (Section 4) restricts software and weights to non-commercial research purposes only; for-profit enterprises may use it only under that non-commercial research limitation.
- AutoModel.from_pretrained(..., trust_remote_code=True) executes remote repository Python code from Hugging Face; pin the commit revision and audit code before running.
- Theia is a visual representation backbone only and does not output safe robot actions directly; downstream policy training and safety envelopes remain the user's responsibility.
- No independent runtime, latency, or downstream robot policy benchmarks were executed by Open Source Robotics.
Compute & fine-tuning
- Inference compute
- Documented hardware: PyTorch / Hugging Face Transformers (CPU or CUDA GPU; README training example uses 8 GPUs via torchrun, while exact inference VRAM and latency are not specified)
- Edge / consumer capable: No
README quick start runs on PyTorch without specifying minimum inference VRAM or latency; edge_capable left false because no edge benchmark is documented in README.
[1] · Editorial assessment
- Fine-tuning
- Supported: Yes
- Methods: multi-teacher distillation training via torchrun train_rvfm.py; downstream robot policy training on extracted Theia features
- Recipe: Upstream fine-tuning documentation
- Host computer
Standard Python/PyTorch host for inference (compact 0.04–0.752 GB safetensors checkpoints); multi-GPU CUDA node for distillation training (README example uses torchrun --nproc_per_node=8).
Inferred from checkpoint sizes and README torchrun training command.
- Operating system
- Linux
Dockerfile and Linux-oriented torchrun workflow documented; standard PyTorch/Transformers inference may run on other OSes.
[1] · Editorial assessment
- Dependencies
- Python >= 3.10
- PyTorch and Hugging Face Transformers
- Webdataset and safetensors (for distillation training)
- Docker (optional)
- Skills
- Python and PyTorch
- Hugging Face Transformers
- Robot policy learning (for downstream control heads)
Inferred from README quick start and training workflow.
[1] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [1]
- Install Python >= 3.10 (or build the repository Dockerfile) and run pip install -e . for local development, or install torch and transformers for Hugging Face inference.
- Load a checkpoint such as theaiinstitute/theia-base-patch16-224-cdiv via AutoModel.from_pretrained(..., trust_remote_code=True) pinned to a reviewed revision.
- Call model.forward_feature(rgb_uint8_tensor) to extract robot-learning representations, or model(rgb_uint8_tensor) for predicted teacher-VFM features.
- For distillation training, convert ImageNet or custom datasets to webdataset shards, extract teacher features, and run torchrun train_rvfm.py.
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| CoRL 2024 Robot Learning & Visual Representation Benchmarks (arXiv:2407.20179) | Upstream reports Theia (theia-<size>-patch16-224-cdiv) outperforms its teacher models and prior robot learning visual models using less training data and smaller model sizes. | Upstream-reported paper abstract summary; exact per-benchmark numerical scores are not included in the supplied README or abstract, and OSR has not independently reproduced evaluations. |
Project notes
Architecture, checkpoints, and usage boundaries
Theia is a visual representation backbone rather than an end-to-end robot policy: it takes 224×224×3 uint8 RGB frames and exposes model.forward_feature() for downstream robot policy heads alongside model() to predict dictionaries of teacher-VFM feature tensors. Six public, ungated safetensors checkpoints are published on Hugging Face across Tiny, Small, and Base DeiT backbones in two teacher configurations: cdiv (CLIP, DINOv2, ViT; 0.04 GB, 0.147 GB, and 0.559 GB) and cddsv (adding SAM and Depth Anything; 0.065 GB, 0.212 GB, and 0.752 GB). [1] [4]
Loading the checkpoints through transformers.AutoModel.from_pretrained(..., trust_remote_code=True) executes custom repository code hosted on Hugging Face, so integrators should pin explicit commit revisions and audit the model code before execution. Exact scalar parameter counts and inference latency figures are not tabulated in the README or arXiv abstract. Training new distilled models requires Python 3.10+, preparing ImageNet or custom image/video datasets into webdataset shards, extracting teacher features, and running torchrun (train_rvfm.py; the README example uses 8 GPUs). [1] [4]
Upstream-reported evaluation
In the CoRL 2024 paper abstract (arXiv:2407.20179), the authors report that Theia (theia-<size>-patch16-224-cdiv) outperforms both its individual teacher models and prior robot-learning visual representations while using less training data and smaller model sizes. Exact per-benchmark numerical scores are not included in the supplied README or abstract. All evaluation claims are upstream-reported; Open Source Robotics has not installed dependencies, trained or fine-tuned models, run inference, or reproduced any robot-learning benchmarks. [1] [5]
What we checked
Source researched only; no installation, training, fine-tuning, inference, benchmark, or physical hardware test performed. No affiliate links or reused media.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Upstream last updated
- Nov 6, 2025 [3]
- Latest release
- Not established
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- commit snapshot; no GitHub release / a63701cdb17d6e1533d623ab72db6cdc788e5eaa · software repository
Separate license matrix
Include the inspectable multi-teacher vision backbone and public safetensors weights under a non-commercial research scope; clearly distinguish RAI Institute's non-commercial research license from permissive open-source terms.
| Layer | Availability / license |
|---|---|
| Code | available · The AI Institute License (custom BSD-style Non-Commercial Research License) Section 4 restricts use to non-commercial research purposes only, including for-profit enterprises. [2] [3] |
| Model weights | available · The AI Institute License (Non-Commercial Research) All 6 Hugging Face safetensors checkpoints are public and ungated, explicitly governed by The AI Institute non-commercial research license. [2] [4] |
| Training datasets | not found · License not fully assessed Pre-training uses external ImageNet (and optional user video datasets) that users must obtain separately under their own terms; repository provides preprocessing scripts and feature_stats/. [1] |
| Documentation | available · The AI Institute License (Non-Commercial Research) README, dataset format doc, and decoding example notebook included in repository. [1] [2] |
Non-commercial research license
Although the repository LICENSE uses a BSD-style header, Section 4 explicitly restricts use to non-commercial research purposes only ("For profit enterprises may use the software, subject to this limitation"), and all six Hugging Face model cards reproduce the same AI Institute License for both the pre-trained model weights and code. External training datasets such as ImageNet are not bundled and retain their own separate terms. Teams needing a commercially permissive vision backbone cannot deploy Theia in production under its public license. [1] [2] [4]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 Theia README
Quick start, installation, distillation training, VFM decoding, and model family overview.Revision, date & where cited
bdaiinstitute/theia contributors · github.com · revision
a63701c· Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 6 more - S002 The AI Institute License (Non-Commercial Research)
Custom BSD-style license restricting use to non-commercial research purposes only.Revision, date & where cited
Robotics and AI Institute LLC dba RAI Institute · github.com · revision
a63701c· Accessed Nov 21, 2026
Used for: Is this for you?, Separate license matrix - S003 GitHub repository metadata and pinned commit
Commit date 2025-11-06; no GitHub releases or tags; not archived.Revision, date & where cited
bdaiinstitute/theia / GitHub · github.com · revision
a63701c· main / · Accessed Nov 21, 2026
Used for: Overview, What we checked, Separate license matrix - S004 Theia Hugging Face collection and model cards
Verified 6 public ungated safetensors checkpoints (tiny/small/base x cdiv/cddsv) with revisions, sizes, and RAI Institute non-commercial license.Revision, date & where cited
The AI Institute / Hugging Face · huggingface.co · 6 safetensors model repos · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 3 more - S005 Theia: Distilling Diverse Vision Foundation Models for Robot Learning (arXiv:2407.20179)
Distills multiple vision foundation models into a compact backbone for robot policy learning; reports outperforming teacher models and prior robot learning models with less data and smaller sizes.Revision, date & where cited
The AI Institute & Stony Brook University (CoRL 2024 / arXiv) · arxiv.org · arXiv:2407.20179v2 abstract · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Upstream-reported benchmarks and 1 more
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile for Theia with 6 verified Hugging Face checkpoints and non-commercial license verification.