Open Source RoboticsRobots, software & learning

Heterogeneous Pre-trained Transformers (HPT)

Robotics model

Modular Stem-Trunk-Head cross-embodiment policy transformer backbone (NeurIPS 2024 Spotlight) pre-trained across 52 heterogeneous datasets with five ungated Hugging Face checkpoints from 3.1M to 226.8M parameters and an MIT-licensed PyTorch codebase. [1] [2] [3] [4]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Cross-embodiment robot learning researchers; Visuomotor policy practitioners fine-tuning on custom robot datasets; Scaling-law researchers in robot policy learning [1] [3]
Parameters
3.1M (HPT-Small), 12.6M (HPT-Base), 50.5M (HPT-Large), 50.6M (HPT-Base-Lang), 226.8M (HPT-XLarge) [1]
Embodiments
Multi-embodiment heterogeneous pretraining across 52 datasets (real-world robots, simulators, deployed robots, human… [1] [3]
Source rights
Some licenses are unresolved. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware
Ecosystem
MuJoCo
License
MIT · license not fully assessed for model weights, datasets · Details by part

Requirements · Research details and dates

On this page

Overview

Heterogeneous Pre-trained Transformers (HPT) is a NeurIPS 2024 Spotlight policy-learning architecture from MIT that pre-trains a shareable transformer trunk across heterogeneous robot embodiments and tasks. Rather than shipping a single monolithic end-to-end VLA, liruiw/HPT provides a modular Stem-Trunk-Head framework and five ungated Hugging Face checkpoints ranging from 3.1M to 226.8M parameters so practitioners can initialize the middle of a policy instead of training from scratch. [1] [3]

Model card summary

Architecture

Modular Stem-Trunk-Head policy transformer: embodiment-specific perception stems align heterogeneous RGB vision and proprioception (plus language in hpt-base-lang) into a short sequence of latent tokens; a shared pre-trained Transformer trunk processes those tokens; and task/embodiment-specific action heads map trunk representations to robot control actions.

[1] [3]

Parameter count

3.1M (HPT-Small), 12.6M (HPT-Base), 50.5M (HPT-Large), 50.6M (HPT-Base-Lang), 226.8M (HPT-XLarge)

Exact published checkpoint sizes from liruiw/HPT README table.

[1]

Input modalities
rgbproprioceptionlanguage

Proprioception and visual inputs across all models; language conditioning supported in HPT-Base (With Language).

[1] [3]

Output modalities
joint-positionsee-pose-deltagripper-command

Modular action heads adapt to embodiment-specific continuous robot action spaces across simulation (Meta-World, MuJoCo) and real-world arms.

[1] [3] · Editorial assessment

Action space

Embodiment- and task-specific continuous action heads attached to the shared HPT transformer trunk (configurable per dataset/environment via modular YAML configs).

[1] [3]

Trained / supported embodiments
  • Multi-embodiment heterogeneous pretraining across 52 datasets (real-world robots, simulators, deployed robots, human video)
  • Meta-World / MuJoCo simulation benchmarks
  • Custom real-robot setups via realrobot_image.yaml

[1] [3]

Training datasets
  • 52 heterogeneous datasets combining large-scale multi-embodiment real-world robot datasets, simulation benchmarks, deployed robot data, and human video datasets (per arXiv:2409.20537v1)

[1] [3]

Interfaces
  • Python module CLI (python -m hpt.run)
  • Hugging Face Hub URI loader (hf://liruiw/hpt-base)
  • Modular YAML configs for stems, trunk, and heads

[1]

Checkpoints

CheckpointFormatSizeAccessPinned revision
liruiw/hpt-xlarge (226.8M)pytorch1.67 GBopen0adad50dbc47
liruiw/hpt-large (50.5M)pytorch0.62 GBopene0d9a3711212
liruiw/hpt-base (12.6M)safetensors0.27 GBopen4586c8c3918b
liruiw/hpt-small (3.1M)pytorch0.09 GBopenfc12980e7655
liruiw/hpt-base-lang (50.6M; HF API resolves repo ID to liruiw/hpt-large-lang)pytorch0.38 GBopen068272250238

Each HF repo includes both full model.pth (decimal GB sizes listed) and trunk.pth (0.01–0.91 GB); liruiw/hpt-base also ships model.safetensors (0.05 GB). Note: querying HF for liruiw/hpt-base-lang resolves to repo ID liruiw/hpt-large-lang.

[1] [4] [5]

Is this for you?

Use cases
  • Fine-tuning a pre-trained HPT trunk on custom single- or multi-task robot datasets via custom stems and heads
  • Benchmarking cross-embodiment transfer on Meta-World and MuJoCo environments
  • Studying policy scaling behavior from 3.1M to 226.8M parameters

[1] [3]

Known limitations
  • Pre-trained trunk checkpoints are designed for downstream transfer learning (replacing or adapting perception stems and action heads) rather than out-of-the-box zero-shot control on arbitrary robots.
  • None of the five Hugging Face checkpoint repositories declare an explicit license YAML tag on the Hub (only the GitHub code repository carries a root MIT LICENSE).
  • Custom datasets require implementing convert_dataset, rollout_runner.py, and learner_trajectory_generator.

[1] [4] [5]

Safety and limitations
  • Pre-trained trunk weights require fine-tuning with task/embodiment-specific stems and heads before deployment.
  • Hugging Face model cards omit explicit license YAML tags, leaving checkpoint license terms formally undeclared on the Hub despite the MIT-licensed GitHub repo.
  • Pretraining spans 52 heterogeneous datasets with distinct upstream data licenses.

[1] [3] [4] [5]

Compute & fine-tuning

Inference compute
  • Documented hardware: CUDA GPU (exact minimum VRAM not stated in README)
  • Precision: fp32
  • Edge / consumer capable: Yes

HPT-Small (3.1M, 0.01 GB trunk.pth) and HPT-Base (12.6M, 0.05 GB safetensors) are lightweight enough for edge deployment, though exact VRAM and latency_ms are not specified.

[1] [4] · Editorial assessment

Fine-tuning
  • Supported: Yes
  • Methods: Transfer learning from pre-trained HPT trunk with modular perception stems and action heads (train.pretrained_dir=hf://liruiw/hpt-base)
  • Recipe: Upstream fine-tuning documentation

[1]

Host computer

GPU with CUDA recommended for training and fine-tuning; README notes pre-training takes substantial compute and recommends starting from published HF checkpoints.

[1]

Operating system
  • Linux

Linux-targeted MuJoCo 2.1.0 EGL and NVIDIA CUDA setup documented in README.

[1] · Editorial assessment

Dependencies
  • Python >= 3.8
  • PyTorch >= 2.0
  • Hydra / YAML configs
  • WandB
  • MuJoCo 2.1.0 (optional, for simulation benchmarks)

[1]

Skills
  • Python
  • PyTorch >= 2.0
  • Hydra configuration
  • WandB

Inferred from CLI flags (+mode=debug, dataset.use_disk=True) and pyproject.toml setup.

[1] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1] · Model card / documentation [1]

  • Clone liruiw/HPT and install with pip install -e . (Python >= 3.8, PyTorch >= 2.0)
  • Optionally install MuJoCo 2.1.0 for MuJoCo/Meta-World simulation benchmarks
  • Review quickstart.ipynb and set train.pretrained_dir to a Hugging Face checkpoint such as hf://liruiw/hpt-base or hf://liruiw/hpt-xlarge
  • Run python -m hpt.run or experiments/scripts/metaworld/train_test_metaworld_20task_finetune.sh

[1]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
Multi-simulator and real-world unseen-task transfer (arXiv:2409.20537v1)>20% improvement in fine-tuned policy performance on unseen tasks across multiple simulator benchmarks and real-world settingsUpstream-reported NeurIPS 2024 Spotlight result after heterogeneous pre-training across 52 datasets; not independently verified by OSR.

[3]

Project notes

Modular Stem-Trunk-Head architecture and five model scales

HPT aligns embodiment-specific vision and proprioception inputs (plus language in the language-conditioned variant) through modular perception stems into a short sequence of latent tokens, processes them with a shared pre-trained Transformer trunk, and maps outputs through task-specific action heads. Five checkpoints are listed in the README: HPT-Small (3.1M, ~0.09 GB / 0.09 GiB model.pth), HPT-Base (12.6M, ~0.27 GB / 0.25 GiB model.pth plus ~0.05 GB model.safetensors), HPT-Large (50.5M, ~0.62 GB / 0.58 GiB), HPT-Base (With Language) (50.6M, ~0.38 GB / 0.35 GiB), and HPT-XLarge (226.8M, ~1.67 GB / 1.56 GiB). Note a naming discrepancy on Hugging Face: the GitHub README links liruiw/hpt-base-lang, whereas Hugging Face API metadata for that URL resolves to repository ID liruiw/hpt-large-lang (068272250238). Each Hub repository includes both full model.pth and standalone trunk.pth weights, loadable via train.pretrained_dir=hf://liruiw/hpt-base. [1] [4] [5]

Fine-tuning workflow and upstream-reported transfer gains

The README documents installation via pip install -e . and lists Python >= 3.8 and PyTorch >= 2.0 (with optional MuJoCo 2.1.0 for Meta-World and MuJoCo benchmarks), alongside an alternative implementation in a LeRobot fork (liruiw/lerobot). Adapting HPT to a custom robot dataset requires implementing a convert_dataset function for training data packing, plus a rollout_runner.py file and a learner_trajectory_generator function for evaluation; perception stem and action head YAML configs (such as realrobot_image.yaml) can be modified as needed. Minimum VRAM and inference latency are not specified in the supplied materials and should not be inferred from checkpoint file sizes. In arXiv:2409.20537v1, the authors report pre-training across 52 heterogeneous datasets (real-world multi-embodiment robot data, simulation, deployed robots, and human video) and achieving over 20% improvement in fine-tuned policy performance on unseen tasks across simulator benchmarks and real-world evaluations. All reported gains are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [3] [4]

What we checked

None. Prepared from public upstream sources without hands-on training, inference, or robot execution.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Dec 6, 2024 [1]
Latest release
Oct 2, 2024 [4]
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
liruiw/HPT@549433f70569917eab84dfe0eae9d84a7c4ad9a3; HF liruiw/hpt-base@4586c8c3918b4b6c9f70de925bf4993568f7d8f4 (and hpt-small/large/xlarge/base-lang) · combined hardware and software repository

Separate license matrix

MIT-licensed codebase and five ungated Hugging Face checkpoints (3.1M–226.8M), classified as partial because the Hugging Face weight repositories omit explicit license metadata and pretraining uses 52 mixed-license external datasets.

LayerAvailability / license
Codeavailable · MIT
[1] [2]
Model weightsavailable · License not fully assessed
All five liruiw/hpt-* Hugging Face repos are ungated and publicly downloadable, but none declare a license field in their HF model card YAML or include a LICENSE file. [4] [5]
Training datasetsavailable · License not fully assessed
Pretrained on a mixture of 52 heterogeneous public/research robot, simulation, and human video datasets with individual upstream licenses. [3]
Documentationavailable · MIT
[1] [2]

Licensing gap on Hub checkpoints

The liruiw/HPT GitHub repository carries a root MIT License (LICENSE at 549433f70569). However, none of the five liruiw/hpt-* Hugging Face repositories declare a license: tag in their model card YAML or include a LICENSE file, and pretraining draws on 52 external datasets with distinct upstream terms. Commercial users should verify checkpoint and dataset rights directly before redistribution. [1] [2] [3] [4] [5]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile drafted from liruiw/HPT@549433f70569 and HF liruiw/hpt-* checkpoints.