RDT-1B (Robotics Diffusion Transformer)
Robotics model1B-parameter Diffusion Transformer foundation model for bimanual and general robot manipulation, pretrained on 1M+ multi-robot episodes with MIT-licensed code, weights, and bimanual fine-tuning dataset. [1] [2] [4]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Robotics researchers; Bimanual manipulation engineers; VLA fine-tuning practitioners [2]
- Source rights
- Named licenses are documented by part. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Overview
RDT-1B (Robotics Diffusion Transformer) is described by upstream as a 1B-parameter Diffusion Transformer policy for bimanual and general manipulation (excluding external frozen vision and language encoders), accompanied by a smaller 170M checkpoint (rdt-170m). Both checkpoints are pretrained on a multi-robot corpus of 46 datasets and conditioned on frozen SigLIP-SO400M-patch14-384 vision features, frozen T5-v1.1-XXL language embeddings, control frequency, and proprioceptive state. [1] [2] [4] [6]
Model card summary
- Architecture
Diffusion Transformer (DiT) action denoiser conditioned on frozen SigLIP-SO400M-patch14-384 vision encoder, frozen T5-v1.1-XXL language encoder, control frequency, and proprioception.
- Base model
google/siglip-so400m-patch14-384 + google/t5-v1_1-xxl (frozen conditioning encoders; DiT pretrained from scratch on robot episodes)
- Parameter count
1B (rdt-1b; ~2.30 GB checkpoint) and 170M (rdt-170m, ~0.31 GB checkpoint), per upstream model card and README; excludes external frozen SigLIP and T5-v1.1-XXL encoders
Checkpoint files are PyTorch .bin (no safetensors parameter count header on Hub).
- Input modalities
- rgblanguageproprioception
Up to 3 RGB camera views (img_history_size: 2), English language instruction, proprioceptive state, and optional control frequency scalar.
- Output modalities
- joint-positionsjoint-velocitiesee-pose-absee-pose-deltagripper-command
Unified 128-D physical action vector covering single- and dual-arm joint positions/velocities, 6D-rotation EEF pose, gripper, and wheeled base velocities.
- Action space
128-D unified physical action space (state_dim: 128 in configs/base.yaml) predicting 64 future action steps (action_chunk_size: 64); uses 6D rotation representation for EEF orientation and maps single-arm robots to the right-arm slots.
- Trained / supported embodiments
- aloha
- mobile-aloha
- AgileX Piper / dual-arm ALOHA
- Multi-robot pretraining embodiments (single-arm, dual-arm, mobile manipulators via 128-D unified vector)
Fine-tuned and deployed on ALOHA dual-arm platform in paper.
- Training datasets
- open-x-embodiment
- droid-dataset
- bridgedata-v2
- rh20t
- RT-1 Dataset
- RoboSet
- rdt-ft-data (6K+ self-collected bimanual episodes, 46 pretraining datasets total)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| rdt-1b (1M-step pretrained) | pytorch | 2.3 GB | open | eb09036cc64c |
| rdt-170m (500K-step small pretrained) | pytorch | 0.31 GB | open | 8aa386cac3bb |
Sizes verified via Hugging Face API blobs=true.
Is this for you?
- Use cases
- Bimanual manipulation policy fine-tuning
- Cross-embodiment manipulation research
- Single-arm or mobile manipulator action chunk prediction
- Known limitations
- Cannot zero-shot generalize to unseen robot embodiments not present in pretraining data
- Requires external frozen SigLIP and T5-v1.1-XXL encoders (T5-v1.1-XXL may exceed RTX 4090 VRAM unless language embeddings are precomputed)
- Safety and limitations
- Does not zero-shot generalize to robot embodiments unseen during pretraining; requires embodiment-specific fine-tuning.
- Requires mapping raw joint/EEF SI units, min-max [0, 1] gripper width, and 6D rotation conventions (configs/state_vec.py, docs/test_6drot.py) into the 128-D unified action vector.
- Frozen T5-v1.1-XXL and SigLIP encoders are separate upstream downloads with their own terms and memory overhead; minimum VRAM with precomputed embeddings and closed-loop latency are unspecified.
Compute & fine-tuning
- Inference compute
- Documented hardware: RTX 4090 (requires precomputed T5 embeddings per README)
- Precision:
bf16 - Edge / consumer capable: No
Exact minimum VRAM and inference latency_ms not stated in inspected README; README explicitly warns RTX 4090 (24 GB) may lack VRAM for live T5-v1.1-XXL encoding.
[6] · Editorial assessment
- Fine-tuning
- Supported: Yes
- Methods: full fine-tune; LoRA / custom HDF5 dataset fine-tuning
- Recipe: Upstream fine-tuning documentation
- Host computer
High-VRAM GPU recommended when loading T5-v1.1-XXL live; RTX 4090 or smaller GPUs require precomputing language embeddings offline (per README).
- Operating system
- Linux
Linux shell and symlink setup documented in README.
[6] · Editorial assessment
- Skills
- Python
- PyTorch
- DeepSpeed / multi-GPU fine-tuning
- HDF5 dataset preparation
Inferred from repository fine-tuning instructions.
[6] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [6] · Model card / documentation [2]
- Clone thu-ml/RoboticsDiffusionTransformer and install dependencies
- Download google/t5-v1_1-xxl and google/siglip-so400m-patch14-384 encoders
- Download robotics-diffusion-transformer/rdt-1b (or rdt-170m) weights from Hugging Face
- Map robot state/action into configs/state_vec.py unified 128-D vector and fine-tune or deploy
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| Real-robot ALOHA dual-arm manipulation suite (arXiv:2410.07864) | Upstream reports SOTA bimanual performance after 6K+ episode fine-tuning (see arXiv:2410.07864 for per-task rates) | Upstream-reported real-robot ALOHA evaluation; not independently verified by OSR. |
Project notes
Unified 128-D state and action representation
The pinned configuration (configs/base.yaml) sets state_dim: 128, action_chunk_size: 64, img_history_size: 2, and num_cameras: 3. Upstream recommends filling the unified vector in SI units without normalizing physical quantities—except gripper width, which is min-max normalized to [0, 1]—using a 6D rotation representation for end-effector orientation and mapping single-arm robots into the right-arm slots (see configs/state_vec.py for slot definitions and docs/test_6drot.py for 6D rotation conversion). Adapting RDT-1B to a new robot requires matching these conventions and fine-tuning on embodiment-specific demonstrations; the base checkpoint does not zero-shot generalize to unseen embodiments. [5] [6]
Compute prerequisites and upstream-reported evaluations
As upstream memory guidance, the README warns that RTX 4090 memory may be insufficient for live T5-v1.1-XXL language encoding during fine-tuning and documents precomputing language embeddings instead: run scripts/encode_lang_batch.py, load the resulting embedding paths through HDF5VLADataset, and pass --precomp_lang_embed in finetune.sh. Minimum VRAM with precomputed embeddings and closed-loop inference latency are unspecified in the inspected material. Upstream's ALOHA dual-arm evaluation results in arXiv:2410.07864 reflect fine-tuning on 6,000+ bimanual episodes and are strictly upstream-reported; Open Source Robotics has not executed training, fine-tuning, inference, or real-robot trials for RDT-1B. [2] [6]
What we checked
None. Prepared from public upstream sources without hands-on hardware execution.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Upstream last updated
- Jan 21, 2026 [1]
- Latest release
- Oct 23, 2024 [6]
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- May 21, 2027
- Reviewed revision
- thu-ml/RoboticsDiffusionTransformer@cd79363a1387e8f81c7724d070ef7e45fd23150f; HF rdt-1b@eb09036cc64ca4945051acbd1bd581d30a1d7711 · software repository
Separate license matrix
Standalone 1B open-weights Diffusion Transformer VLA model with verified MIT code, MIT weights, and MIT fine-tuning dataset (rdt-ft-data).
| Layer | Availability / license |
|---|---|
| Code | available · MIT MIT License (Copyright 2024 TSAIL group) verified in root LICENSE. [1] [3] |
| Model weights | available · MIT Both rdt-1b and rdt-170m model cards declare license: mit; external frozen SigLIP and T5-v1.1-XXL encoders are separate upstream downloads. [2] [4] |
| Training datasets | available · MIT (rdt-ft-data fine-tuning dataset); separate upstream terms across 46 pretraining datasets (unverified individually in this review) Self-collected rdt-ft-data (648 GB) declares MIT; the 46 pretraining datasets carry separate upstream licenses not individually excerpted in this review. [2] [4] |
| Documentation | available · MIT [3] [6] |
Separate code, weights, encoder, and dataset licenses
The GitHub repository, both Hugging Face checkpoints (rdt-1b and rdt-170m, PyTorch .bin), and the self-collected bimanual fine-tuning dataset rdt-ft-data declare MIT licensing. However, the external frozen SigLIP and T5-v1.1-XXL encoders are downloaded separately under their own upstream terms, and the 46 pretraining datasets (including Open X-Embodiment, DROID, BridgeData V2, and RH20T) carry separate upstream licenses not individually excerpted in this review and requiring independent verification. [2] [3] [4]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 thu-ml/RoboticsDiffusionTransformer GitHub metadata and HEAD commit
Default branch main HEAD is cd79363a1387e8f81c7724d070ef7e45fd23150f (2026-01-21T09:48:32Z); SPDX license MIT.Revision, date & where cited
thu-ml (GitHub) · github.com · revision
cd79363· GitHub API repos/thu-ml/RoboticsDiffusionTransformer & commits/main · Accessed Nov 21, 2026
Used for: Overview, What we checked, Separate license matrix - S002 Hugging Face model card: robotics-diffusion-transformer/rdt-1b
Confirms ungated MIT-licensed checkpoint (2.30 GB total, PyTorch .bin), SigLIP-SO400M + T5-v1.1-XXL encoders, 46 pretraining datasets, and 64-step action prediction.Revision, date & where cited
robotics-diffusion-transformer (Hugging Face) · huggingface.co · HF API & README.md at revision eb09036cc64c · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 4 more - S003 thu-ml/RoboticsDiffusionTransformer LICENSE
MIT License, Copyright (c) 2024 TSAIL group.Revision, date & where cited
TSAIL group · github.com · revision
cd79363· LICENSE at commit cd79363a1387 · Accessed Nov 21, 2026
Used for: Separate license matrix - S004 Hugging Face API metadata for rdt-170m and rdt-ft-data
rdt-170m @ 8aa386cac3bb (0.31 GB, ungated, MIT); rdt-ft-data @ a3f4b6624d4f (648.0 GB, ungated, MIT).Revision, date & where cited
robotics-diffusion-transformer (Hugging Face) · huggingface.co · HF API /api/models/robotics-diffusion-transformer/rdt-170m & /api/datasets/robotics-diffusion-transformer/rdt-ft-data · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Separate license matrix - S005 RDT-1B configs/base.yaml at pinned commit
img_history_size: 2, action_chunk_size: 64, num_cameras: 3, state_dim: 128.Revision, date & where cited
thu-ml (GitHub) · github.com · revision
cd79363· configs/base.yaml lines 1-12 · Accessed Nov 21, 2026
Used for: Model card summary, Unified 128-D state and action representation - S006 thu-ml/RoboticsDiffusionTransformer README.md at pinned commit
Documents ALOHA fine-tuning, 6D EEF rotation convention, RTX 4090 T5 embedding precomputation requirement, and RDT-170M release.Revision, date & where cited
thu-ml (GitHub) · github.com · revision
cd79363· README.md at commit cd79363a1387 · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Compute & fine-tuning and 6 more
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model draft prepared from pinned GitHub commit cd79363a1387 and HF revision eb09036cc64c.