Open Source RoboticsRobots, software & learning

RDT-1B (Robotics Diffusion Transformer)

Robotics model

1B-parameter Diffusion Transformer foundation model for bimanual and general robot manipulation, pretrained on 1M+ multi-robot episodes with MIT-licensed code, weights, and bimanual fine-tuning dataset. [1] [2] [4]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Robotics researchers; Bimanual manipulation engineers; VLA fine-tuning practitioners [2]
Parameters
1B (rdt-1b [2] [4] [6]
Embodiments
aloha, mobile-aloha, AgileX Piper / dual-arm ALOHA, … [2] [6]
Source rights
Named licenses are documented by part. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware

Requirements · Research details and dates

On this page

Overview

RDT-1B (Robotics Diffusion Transformer) is described by upstream as a 1B-parameter Diffusion Transformer policy for bimanual and general manipulation (excluding external frozen vision and language encoders), accompanied by a smaller 170M checkpoint (rdt-170m). Both checkpoints are pretrained on a multi-robot corpus of 46 datasets and conditioned on frozen SigLIP-SO400M-patch14-384 vision features, frozen T5-v1.1-XXL language embeddings, control frequency, and proprioceptive state. [1] [2] [4] [6]

Model card summary

Architecture

Diffusion Transformer (DiT) action denoiser conditioned on frozen SigLIP-SO400M-patch14-384 vision encoder, frozen T5-v1.1-XXL language encoder, control frequency, and proprioception.

[2] [6]

Base model

google/siglip-so400m-patch14-384 + google/t5-v1_1-xxl (frozen conditioning encoders; DiT pretrained from scratch on robot episodes)

[2]

Parameter count

1B (rdt-1b; ~2.30 GB checkpoint) and 170M (rdt-170m, ~0.31 GB checkpoint), per upstream model card and README; excludes external frozen SigLIP and T5-v1.1-XXL encoders

Checkpoint files are PyTorch .bin (no safetensors parameter count header on Hub).

[2] [4] [6]

Input modalities
rgblanguageproprioception

Up to 3 RGB camera views (img_history_size: 2), English language instruction, proprioceptive state, and optional control frequency scalar.

[2] [5]

Output modalities
joint-positionsjoint-velocitiesee-pose-absee-pose-deltagripper-command

Unified 128-D physical action vector covering single- and dual-arm joint positions/velocities, 6D-rotation EEF pose, gripper, and wheeled base velocities.

[2] [5]

Action space

128-D unified physical action space (state_dim: 128 in configs/base.yaml) predicting 64 future action steps (action_chunk_size: 64); uses 6D rotation representation for EEF orientation and maps single-arm robots to the right-arm slots.

[2] [5] [6]

Trained / supported embodiments
  • aloha
  • mobile-aloha
  • AgileX Piper / dual-arm ALOHA
  • Multi-robot pretraining embodiments (single-arm, dual-arm, mobile manipulators via 128-D unified vector)

Fine-tuned and deployed on ALOHA dual-arm platform in paper.

[2] [6]

Training datasets

[2] [4]

Interfaces
  • Python API
  • Hugging Face Hub checkpoint
  • HDF5 dataset loader

[2] [6]

Checkpoints

CheckpointFormatSizeAccessPinned revision
rdt-1b (1M-step pretrained)pytorch2.3 GBopeneb09036cc64c
rdt-170m (500K-step small pretrained)pytorch0.31 GBopen8aa386cac3bb

Sizes verified via Hugging Face API blobs=true.

[2] [4]

Is this for you?

Use cases
  • Bimanual manipulation policy fine-tuning
  • Cross-embodiment manipulation research
  • Single-arm or mobile manipulator action chunk prediction

[2]

Known limitations
  • Cannot zero-shot generalize to unseen robot embodiments not present in pretraining data
  • Requires external frozen SigLIP and T5-v1.1-XXL encoders (T5-v1.1-XXL may exceed RTX 4090 VRAM unless language embeddings are precomputed)

[2] [6]

Safety and limitations
  • Does not zero-shot generalize to robot embodiments unseen during pretraining; requires embodiment-specific fine-tuning.
  • Requires mapping raw joint/EEF SI units, min-max [0, 1] gripper width, and 6D rotation conventions (configs/state_vec.py, docs/test_6drot.py) into the 128-D unified action vector.
  • Frozen T5-v1.1-XXL and SigLIP encoders are separate upstream downloads with their own terms and memory overhead; minimum VRAM with precomputed embeddings and closed-loop latency are unspecified.

[2] [6]

Compute & fine-tuning

Inference compute
  • Documented hardware: RTX 4090 (requires precomputed T5 embeddings per README)
  • Precision: bf16
  • Edge / consumer capable: No

Exact minimum VRAM and inference latency_ms not stated in inspected README; README explicitly warns RTX 4090 (24 GB) may lack VRAM for live T5-v1.1-XXL encoding.

[6] · Editorial assessment

Fine-tuning

[6]

Host computer

High-VRAM GPU recommended when loading T5-v1.1-XXL live; RTX 4090 or smaller GPUs require precomputing language embeddings offline (per README).

[6]

Operating system
  • Linux

Linux shell and symlink setup documented in README.

[6] · Editorial assessment

Dependencies
  • Python
  • PyTorch
  • Transformers
  • google/siglip-so400m-patch14-384
  • google/t5-v1_1-xxl

[2] [6]

Skills
  • Python
  • PyTorch
  • DeepSpeed / multi-GPU fine-tuning
  • HDF5 dataset preparation

Inferred from repository fine-tuning instructions.

[6] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [6] · Model card / documentation [2]

  • Clone thu-ml/RoboticsDiffusionTransformer and install dependencies
  • Download google/t5-v1_1-xxl and google/siglip-so400m-patch14-384 encoders
  • Download robotics-diffusion-transformer/rdt-1b (or rdt-170m) weights from Hugging Face
  • Map robot state/action into configs/state_vec.py unified 128-D vector and fine-tune or deploy

[6]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
Real-robot ALOHA dual-arm manipulation suite (arXiv:2410.07864)Upstream reports SOTA bimanual performance after 6K+ episode fine-tuning (see arXiv:2410.07864 for per-task rates)Upstream-reported real-robot ALOHA evaluation; not independently verified by OSR.

[6]

Project notes

Unified 128-D state and action representation

The pinned configuration (configs/base.yaml) sets state_dim: 128, action_chunk_size: 64, img_history_size: 2, and num_cameras: 3. Upstream recommends filling the unified vector in SI units without normalizing physical quantities—except gripper width, which is min-max normalized to [0, 1]—using a 6D rotation representation for end-effector orientation and mapping single-arm robots into the right-arm slots (see configs/state_vec.py for slot definitions and docs/test_6drot.py for 6D rotation conversion). Adapting RDT-1B to a new robot requires matching these conventions and fine-tuning on embodiment-specific demonstrations; the base checkpoint does not zero-shot generalize to unseen embodiments. [5] [6]

Compute prerequisites and upstream-reported evaluations

As upstream memory guidance, the README warns that RTX 4090 memory may be insufficient for live T5-v1.1-XXL language encoding during fine-tuning and documents precomputing language embeddings instead: run scripts/encode_lang_batch.py, load the resulting embedding paths through HDF5VLADataset, and pass --precomp_lang_embed in finetune.sh. Minimum VRAM with precomputed embeddings and closed-loop inference latency are unspecified in the inspected material. Upstream's ALOHA dual-arm evaluation results in arXiv:2410.07864 reflect fine-tuning on 6,000+ bimanual episodes and are strictly upstream-reported; Open Source Robotics has not executed training, fine-tuning, inference, or real-robot trials for RDT-1B. [2] [6]

What we checked

None. Prepared from public upstream sources without hands-on hardware execution.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Jan 21, 2026 [1]
Latest release
Oct 23, 2024 [6]
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
May 21, 2027
Reviewed revision
thu-ml/RoboticsDiffusionTransformer@cd79363a1387e8f81c7724d070ef7e45fd23150f; HF rdt-1b@eb09036cc64ca4945051acbd1bd581d30a1d7711 · software repository

Separate license matrix

Standalone 1B open-weights Diffusion Transformer VLA model with verified MIT code, MIT weights, and MIT fine-tuning dataset (rdt-ft-data).

LayerAvailability / license
Codeavailable · MIT
MIT License (Copyright 2024 TSAIL group) verified in root LICENSE. [1] [3]
Model weightsavailable · MIT
Both rdt-1b and rdt-170m model cards declare license: mit; external frozen SigLIP and T5-v1.1-XXL encoders are separate upstream downloads. [2] [4]
Training datasetsavailable · MIT (rdt-ft-data fine-tuning dataset); separate upstream terms across 46 pretraining datasets (unverified individually in this review)
Self-collected rdt-ft-data (648 GB) declares MIT; the 46 pretraining datasets carry separate upstream licenses not individually excerpted in this review. [2] [4]
Documentationavailable · MIT
[3] [6]

Separate code, weights, encoder, and dataset licenses

The GitHub repository, both Hugging Face checkpoints (rdt-1b and rdt-170m, PyTorch .bin), and the self-collected bimanual fine-tuning dataset rdt-ft-data declare MIT licensing. However, the external frozen SigLIP and T5-v1.1-XXL encoders are downloaded separately under their own upstream terms, and the 46 pretraining datasets (including Open X-Embodiment, DROID, BridgeData V2, and RH20T) carry separate upstream licenses not individually excerpted in this review and requiring independent verification. [2] [3] [4]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model draft prepared from pinned GitHub commit cd79363a1387 and HF revision eb09036cc64c.