CrossFormer
Robotics modelMIT-licensed 130M-parameter JAX/Flax cross-embodiment Transformer policy from UC Berkeley RAIL trained on 900K trajectories across 20 robot embodiments (single-arm, bimanual, wheeled navigation, quadruped, and quadcopter), with an ungated MIT Orbax checkpoint on Hugging Face. [1] [2] [3] [4] [5]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Cross-embodiment robot learning researchers; JAX / Octo ecosystem practitioners; Multi-platform robotics labs (arms, mobile bases, quadrupeds) [1]
- Embodiments
- Single-arm manipulators (e.g., WidowX/Bridge, Franka/DROID, UR5, KUKA, Jaco), Dual-arm / bimanual manipulators… [1] [4]
- Source rights
- Named licenses are documented by part. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
Overview
CrossFormer (rail-berkeley/crossformer) is a 130M-parameter JAX/Flax Transformer robot policy from UC Berkeley RAIL trained on 900K trajectories across 20 robot embodiments. Built directly on the Octo codebase, it ships as a single ungated MIT-licensed Orbax checkpoint that upstream reports as capable of controlling single-arm and bimanual manipulators, wheeled navigation robots, quadrupeds, and quadcopters from language commands or goal images. [1] [2] [3] [5]
Model card summary
- Architecture
Modular cross-embodiment Transformer policy built on the Octo JAX/Flax codebase (window_size: 5), with seven embodiment/camera observation tokenizers (primary, high, left, right, nav, bimanual, quadruped) and four action readout heads (single_arm, bimanual, nav, quadruped), conditioned on language instructions or goal images.
- Parameter count
130M parameters (per GitHub README; Orbax checkpoint 300000/default/checkpoint is 0.52 GB / 0.48 GiB)
- Input modalities
- rgbproprioceptionlanguage
Supports multi-camera RGB (primary, high, left, right, nav), embodiment-specific proprioception, and language or goal-image task conditioning.
- Output modalities
- ee-pose-deltajoint-positionsgripper-commandwaypoints
Four separate heads verified in config.json: single_arm (7-D x 4), bimanual (14-D x 100), nav (2-D x 4), quadruped (12-D x 1).
- Action space
Multi-head continuous action chunks per config.json: single_arm (action_dim: 7, action_horizon: 4), bimanual (action_dim: 14, action_horizon: 100), nav (action_dim: 2, action_horizon: 4), and quadruped (action_dim: 12, action_horizon: 1).
- Trained / supported embodiments
- Single-arm manipulators (e.g., WidowX/Bridge, Franka/DROID, UR5, KUKA, Jaco)
- Dual-arm / bimanual manipulators (ALOHA)
- Wheeled navigation robots (GNM / OmniMimic)
- Quadrupeds (Unitree Go1)
- Quadcopters (per README)
- Training datasets
- 900K trajectories across 20 robot embodiments (35 dataset keys in dataset_statistics.json including Open X-Embodiment subsets, BridgeData V2, DROID, ALOHA datasets, Go1, and OmniMimic GNM)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| rail-berkeley/crossformer (step 300000) | jax-orbax | 0.52 GB | open | c7dea2691aed |
Ungated MIT JAX/Orbax checkpoint (300000/default/checkpoint, 519,021,888 bytes = 0.52 GB / 0.48 GiB) verified via Hugging Face API.
Is this for you?
- Use cases
- Zero-shot or fine-tuned control of single-arm, bimanual, wheeled, quadruped, and quadcopter platforms
- Fine-tuning with new observation tokenizers or action heads on custom RLDS datasets
- Remote GPU inference hosting via scripts/server.py for onboard-compute-limited robots
- Known limitations
- Pinned to legacy jax==0.4.20 (CUDA 11 pip wheels or TPU wheels), which may require adaptation on newer CUDA/GPU stacks.
- Full pre-training reproduction requires 47 hours on a TPUv5-256 pod and RLDS conversion of Open X-Embodiment and multi-embodiment datasets.
- Repository has been dormant since August 2024.
Compute & fine-tuning
- Inference compute
- Documented hardware: Single NVIDIA GPU (or remote GPU via scripts/server.py)
- Precision:
fp32 - Edge / consumer capable: Yes
0.48 GB 130M-parameter Orbax checkpoint fits on consumer GPUs; README also provides scripts/server.py for remote inference on compute-constrained robots.
- Fine-tuning
- Supported: Yes
- Methods: Full fine-tuning on RLDS datasets via scripts/finetune.py; Frozen-backbone / action-head-only fine-tuning; New observation tokenizer and/or action head initialization for novel morphologies
- Recipe: Upstream fine-tuning documentation
- Host computer
Pre-training takes 47 hours on a TPUv5-256 pod; fine-tuning and inference target accessible single-GPU or remote-server GPU setups (exact minimum VRAM not stated in README).
- Operating system
- Linux
Standard Linux conda + JAX CUDA/TPU environment documented.
[1] · Editorial assessment
- Dependencies
- Python 3.10
- JAX 0.4.20 (CUDA 11 or TPU)
- Flax / Orbax
- rlds_dataset_mod / rlds_dataset_builder (for pretraining/finetuning)
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [1]
- Create a Python 3.10 conda environment and run pip install -e . and pip install -r requirements.txt
- Install jax[cuda11_pip]==0.4.20 for GPU or jax[tpu]==0.4.20 for TPU
- Load the pretrained checkpoint via CrossFormerModel.load_pretrained('hf://rail-berkeley/crossformer')
- Run scripts/server.py for remote inference or convert custom datasets to RLDS and run scripts/finetune.py
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| Cross-embodiment real-robot evaluation across manipulation, navigation, locomotion, and aviation (arXiv:2408.11812v1) | See arXiv:2408.11812v1 (single 130M policy controlling single/dual-arm manipulation, wheeled robots, quadrupeds, and quadcopters) | Upstream-reported multi-embodiment claims in arXiv:2408.11812v1; exact per-platform task tables not independently verified by OSR. |
Project notes
Multi-head architecture, embodiment tokenizers, and RLDS fine-tuning
At the pinned Hugging Face revision (c7dea2691aed), config.json defines a modular attention Transformer (window_size: 5) with seven observation tokenizers (primary, high, left, right, nav, bimanual, quadruped) and four separate continuous action readout heads: single_arm (action_dim: 7, action_horizon: 4), bimanual (action_dim: 14, action_horizon: 100), nav (action_dim: 2, action_horizon: 4), and quadruped (action_dim: 12, action_horizon: 1). While upstream (arXiv:2408.11812v1 and README) reports out-of-the-box quadcopter control, how aviation actions map to those four published checkpoint heads is not established by the inspected config.json. The checkpoint (300000/default/checkpoint, ~0.52 GB / 0.48 GiB) loads via CrossFormerModel.load_pretrained("hf://rail-berkeley/crossformer"). To fine-tune on custom data, the README documents converting target trajectories to RLDS via rlds_dataset_builder, editing scripts/configs/finetune_config.py, and running scripts/finetune.py --config scripts/configs/finetune_config.py. Datasets matching a pretrained observation and action space can reuse the existing tokenizers and action heads; otherwise users can initialize new observation tokenizers or action heads while retaining the pretrained transformer backbone (with options to fine-tune the full model or freeze the transformer). [1] [3] [4] [5]
Deployment footprint and upstream-reported scope
Pre-training reproduction takes 47 hours on a TPUv5-256 pod over Open X-Embodiment and multi-embodiment datasets (35 dataset keys verified in dataset_statistics.json, including BridgeData V2, DROID, ALOHA, Go1, and OmniMimic GNM). For deployment, the repository provides scripts/server.py for remote GPU inference alongside a Colab notebook, though installation pins legacy jax==0.4.20 (CUDA 11 or TPU wheels) and the repo has been dormant since August 2024. Minimum inference VRAM and per-embodiment latency are not specified in the inspected materials. All multi-embodiment evaluation claims in arXiv:2408.11812v1 are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [3] [4] [5]
What we checked
None. Prepared from public upstream sources without hands-on training, inference, or robot execution.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Latest release
- Aug 26, 2024 [3]
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- rail-berkeley/crossformer@4a56b64411e7ec039ea6cce6bbbe65a38f677db1; HF rail-berkeley/crossformer@c7dea2691aed3656537c5126a0a77df84a28abd7 · combined hardware and software repository
Separate license matrix
MIT-licensed 130M cross-embodiment policy codebase and ungated MIT Hugging Face Orbax checkpoint with documented RLDS pretraining and fine-tuning recipes.
| Layer | Availability / license |
|---|---|
| Code | available · MIT [1] [2] |
| Model weights | available · MIT Model card YAML declares license: mit. [3] |
| Training datasets | available · Mixed (Open X-Embodiment and multi-embodiment datasets; individual upstream licenses apply) 900K trajectories across 20 embodiments (35 dataset keys verified in dataset_statistics.json). [1] [4] |
| Documentation | available · MIT [1] [2] |
MIT code and weights over a mixed-license data mixture
Both the GitHub repository (LICENSE at 4a56b64411e7) and the Hugging Face model repository (c7dea2691aed, YAML license: mit) are permissively licensed under MIT. The 900K-trajectory pretraining mixture combines 35 upstream robotics datasets whose individual licenses vary by contributor and should be reviewed separately when reproducing pretraining. [1] [2] [3] [4]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 rail-berkeley/crossformer GitHub repository and README.md
Documents 130M checkpoint, 900K trajectories across 20 embodiments, Octo-derived JAX 0.4.20 codebase, 47h TPUv5-256 pretraining, scripts/server.py, and RLDS fine-tuning.Revision, date & where cited
UC Berkeley RAIL (GitHub) · github.com · revision
4a56b64· README.md at commit 4a56b64411e7 (2024-08-26) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 7 more - S002 rail-berkeley/crossformer LICENSE
MIT License, Copyright (c) 2023 Robotic AI & Learning Lab Berkeley.Revision, date & where cited
Robotic AI & Learning Lab Berkeley · github.com · revision
4a56b64· LICENSE at commit 4a56b64411e7 · Accessed Nov 21, 2026
Used for: Overview, Separate license matrix - S003 Hugging Face model repository: rail-berkeley/crossformer
Ungated MIT checkpoint (300000/default/checkpoint, 0.52 GB / 0.48 GiB JAX/Orbax), config.json, dataset_statistics.json.Revision, date & where cited
UC Berkeley RAIL (Hugging Face) · huggingface.co · HF API & README.md at revision c7dea2691aed (2024-08-26) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Compute & fine-tuning and 4 more - S004 rail-berkeley/crossformer config.json and dataset_statistics.json
Verifies window_size: 5, 7 observation tokenizers, 4 readout heads (single_arm 7x4, bimanual 14x100, nav 2x4, quadruped 12x1), and 35 dataset statistics keys.Revision, date & where cited
UC Berkeley RAIL (Hugging Face) · huggingface.co · config.json & dataset_statistics.json at revision c7dea2691aed · Accessed Nov 21, 2026
Used for: Model card summary, Is this for you?, Multi-head architecture, embodiment tokenizers, and RLDS fine-tuning, Deployment footprint and upstream-reported scope and 1 more - S005 Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviation
Describes training a single cross-embodied transformer policy across manipulation, navigation, locomotion, and aviation platforms.Revision, date & where cited
arXiv (2408.11812v1) · arxiv.org · arXiv abstract 2408.11812v1 · Accessed Nov 21, 2026
Used for: Overview, Upstream-reported benchmarks, Multi-head architecture, embodiment tokenizers, and RLDS fine-tuning, Deployment footprint and upstream-reported scope
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile drafted from rail-berkeley/crossformer@4a56b64411e7 and HF c7dea2691aed.