Open Source RoboticsRobots, software & learning

CrossFormer

Robotics model

MIT-licensed 130M-parameter JAX/Flax cross-embodiment Transformer policy from UC Berkeley RAIL trained on 900K trajectories across 20 robot embodiments (single-arm, bimanual, wheeled navigation, quadruped, and quadcopter), with an ungated MIT Orbax checkpoint on Hugging Face. [1] [2] [3] [4] [5]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Cross-embodiment robot learning researchers; JAX / Octo ecosystem practitioners; Multi-platform robotics labs (arms, mobile bases, quadrupeds) [1]
Parameters
130M parameters (per GitHub README [1] [3]
Embodiments
Single-arm manipulators (e.g., WidowX/Bridge, Franka/DROID, UR5, KUKA, Jaco), Dual-arm / bimanual manipulators… [1] [4]
Source rights
Named licenses are documented by part. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware

Requirements · Research details and dates

On this page

Overview

CrossFormer (rail-berkeley/crossformer) is a 130M-parameter JAX/Flax Transformer robot policy from UC Berkeley RAIL trained on 900K trajectories across 20 robot embodiments. Built directly on the Octo codebase, it ships as a single ungated MIT-licensed Orbax checkpoint that upstream reports as capable of controlling single-arm and bimanual manipulators, wheeled navigation robots, quadrupeds, and quadcopters from language commands or goal images. [1] [2] [3] [5]

Model card summary

Architecture

Modular cross-embodiment Transformer policy built on the Octo JAX/Flax codebase (window_size: 5), with seven embodiment/camera observation tokenizers (primary, high, left, right, nav, bimanual, quadruped) and four action readout heads (single_arm, bimanual, nav, quadruped), conditioned on language instructions or goal images.

[1] [4]

Parameter count

130M parameters (per GitHub README; Orbax checkpoint 300000/default/checkpoint is 0.52 GB / 0.48 GiB)

[1] [3]

Input modalities
rgbproprioceptionlanguage

Supports multi-camera RGB (primary, high, left, right, nav), embodiment-specific proprioception, and language or goal-image task conditioning.

[1] [4]

Output modalities
ee-pose-deltajoint-positionsgripper-commandwaypoints

Four separate heads verified in config.json: single_arm (7-D x 4), bimanual (14-D x 100), nav (2-D x 4), quadruped (12-D x 1).

[1] [4]

Action space

Multi-head continuous action chunks per config.json: single_arm (action_dim: 7, action_horizon: 4), bimanual (action_dim: 14, action_horizon: 100), nav (action_dim: 2, action_horizon: 4), and quadruped (action_dim: 12, action_horizon: 1).

[4]

Trained / supported embodiments
  • Single-arm manipulators (e.g., WidowX/Bridge, Franka/DROID, UR5, KUKA, Jaco)
  • Dual-arm / bimanual manipulators (ALOHA)
  • Wheeled navigation robots (GNM / OmniMimic)
  • Quadrupeds (Unitree Go1)
  • Quadcopters (per README)

[1] [4]

Training datasets
  • 900K trajectories across 20 robot embodiments (35 dataset keys in dataset_statistics.json including Open X-Embodiment subsets, BridgeData V2, DROID, ALOHA datasets, Go1, and OmniMimic GNM)

[1] [4]

Interfaces
  • Python API (CrossFormerModel.load_pretrained)
  • Remote inference server (scripts/server.py)
  • JAX/Orbax checkpoints on Hugging Face Hub
  • RLDS datasets

[1] [3]

Checkpoints

CheckpointFormatSizeAccessPinned revision
rail-berkeley/crossformer (step 300000)jax-orbax0.52 GBopenc7dea2691aed

Ungated MIT JAX/Orbax checkpoint (300000/default/checkpoint, 519,021,888 bytes = 0.52 GB / 0.48 GiB) verified via Hugging Face API.

[1] [3]

Is this for you?

Use cases
  • Zero-shot or fine-tuned control of single-arm, bimanual, wheeled, quadruped, and quadcopter platforms
  • Fine-tuning with new observation tokenizers or action heads on custom RLDS datasets
  • Remote GPU inference hosting via scripts/server.py for onboard-compute-limited robots

[1]

Known limitations
  • Pinned to legacy jax==0.4.20 (CUDA 11 pip wheels or TPU wheels), which may require adaptation on newer CUDA/GPU stacks.
  • Full pre-training reproduction requires 47 hours on a TPUv5-256 pod and RLDS conversion of Open X-Embodiment and multi-embodiment datasets.
  • Repository has been dormant since August 2024.

[1]

Safety and limitations
  • Zero-shot control is scoped to embodiments and sensor setups represented in the 20-embodiment pretraining mixture; novel morphologies require RLDS fine-tuning.
  • Pretraining mixture combines 35 external datasets whose individual licenses and data quality vary.

[1] [4]

Compute & fine-tuning

Inference compute
  • Documented hardware: Single NVIDIA GPU (or remote GPU via scripts/server.py)
  • Precision: fp32
  • Edge / consumer capable: Yes

0.48 GB 130M-parameter Orbax checkpoint fits on consumer GPUs; README also provides scripts/server.py for remote inference on compute-constrained robots.

[1] [3] · Editorial assessment

Fine-tuning
  • Supported: Yes
  • Methods: Full fine-tuning on RLDS datasets via scripts/finetune.py; Frozen-backbone / action-head-only fine-tuning; New observation tokenizer and/or action head initialization for novel morphologies
  • Recipe: Upstream fine-tuning documentation

[1]

Host computer

Pre-training takes 47 hours on a TPUv5-256 pod; fine-tuning and inference target accessible single-GPU or remote-server GPU setups (exact minimum VRAM not stated in README).

[1]

Operating system
  • Linux

Standard Linux conda + JAX CUDA/TPU environment documented.

[1] · Editorial assessment

Dependencies
  • Python 3.10
  • JAX 0.4.20 (CUDA 11 or TPU)
  • Flax / Orbax
  • rlds_dataset_mod / rlds_dataset_builder (for pretraining/finetuning)

[1]

Skills
  • Python 3.10
  • JAX / Flax / Orbax
  • RLDS dataset preparation

Inferred from JAX installation, Orbax checkpoint format, and rlds_dataset_builder instructions.

[1] [3] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1] · Model card / documentation [1]

  • Create a Python 3.10 conda environment and run pip install -e . and pip install -r requirements.txt
  • Install jax[cuda11_pip]==0.4.20 for GPU or jax[tpu]==0.4.20 for TPU
  • Load the pretrained checkpoint via CrossFormerModel.load_pretrained('hf://rail-berkeley/crossformer')
  • Run scripts/server.py for remote inference or convert custom datasets to RLDS and run scripts/finetune.py

[1]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
Cross-embodiment real-robot evaluation across manipulation, navigation, locomotion, and aviation (arXiv:2408.11812v1)See arXiv:2408.11812v1 (single 130M policy controlling single/dual-arm manipulation, wheeled robots, quadrupeds, and quadcopters)Upstream-reported multi-embodiment claims in arXiv:2408.11812v1; exact per-platform task tables not independently verified by OSR.

[1] [5]

Project notes

Multi-head architecture, embodiment tokenizers, and RLDS fine-tuning

At the pinned Hugging Face revision (c7dea2691aed), config.json defines a modular attention Transformer (window_size: 5) with seven observation tokenizers (primary, high, left, right, nav, bimanual, quadruped) and four separate continuous action readout heads: single_arm (action_dim: 7, action_horizon: 4), bimanual (action_dim: 14, action_horizon: 100), nav (action_dim: 2, action_horizon: 4), and quadruped (action_dim: 12, action_horizon: 1). While upstream (arXiv:2408.11812v1 and README) reports out-of-the-box quadcopter control, how aviation actions map to those four published checkpoint heads is not established by the inspected config.json. The checkpoint (300000/default/checkpoint, ~0.52 GB / 0.48 GiB) loads via CrossFormerModel.load_pretrained("hf://rail-berkeley/crossformer"). To fine-tune on custom data, the README documents converting target trajectories to RLDS via rlds_dataset_builder, editing scripts/configs/finetune_config.py, and running scripts/finetune.py --config scripts/configs/finetune_config.py. Datasets matching a pretrained observation and action space can reuse the existing tokenizers and action heads; otherwise users can initialize new observation tokenizers or action heads while retaining the pretrained transformer backbone (with options to fine-tune the full model or freeze the transformer). [1] [3] [4] [5]

Deployment footprint and upstream-reported scope

Pre-training reproduction takes 47 hours on a TPUv5-256 pod over Open X-Embodiment and multi-embodiment datasets (35 dataset keys verified in dataset_statistics.json, including BridgeData V2, DROID, ALOHA, Go1, and OmniMimic GNM). For deployment, the repository provides scripts/server.py for remote GPU inference alongside a Colab notebook, though installation pins legacy jax==0.4.20 (CUDA 11 or TPU wheels) and the repo has been dormant since August 2024. Minimum inference VRAM and per-embodiment latency are not specified in the inspected materials. All multi-embodiment evaluation claims in arXiv:2408.11812v1 are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [3] [4] [5]

What we checked

None. Prepared from public upstream sources without hands-on training, inference, or robot execution.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Aug 26, 2024 [1] [3]
Latest release
Aug 26, 2024 [3]
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
rail-berkeley/crossformer@4a56b64411e7ec039ea6cce6bbbe65a38f677db1; HF rail-berkeley/crossformer@c7dea2691aed3656537c5126a0a77df84a28abd7 · combined hardware and software repository

Separate license matrix

MIT-licensed 130M cross-embodiment policy codebase and ungated MIT Hugging Face Orbax checkpoint with documented RLDS pretraining and fine-tuning recipes.

LayerAvailability / license
Codeavailable · MIT
[1] [2]
Model weightsavailable · MIT
Model card YAML declares license: mit. [3]
Training datasetsavailable · Mixed (Open X-Embodiment and multi-embodiment datasets; individual upstream licenses apply)
900K trajectories across 20 embodiments (35 dataset keys verified in dataset_statistics.json). [1] [4]
Documentationavailable · MIT
[1] [2]

MIT code and weights over a mixed-license data mixture

Both the GitHub repository (LICENSE at 4a56b64411e7) and the Hugging Face model repository (c7dea2691aed, YAML license: mit) are permissively licensed under MIT. The 900K-trajectory pretraining mixture combines 35 upstream robotics datasets whose individual licenses vary by contributor and should be reviewed separately when reproducing pretraining. [1] [2] [3] [4]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile drafted from rail-berkeley/crossformer@4a56b64411e7 and HF c7dea2691aed.