Heterogeneous Pre-trained Transformers (HPT)
Robotics modelModular Stem-Trunk-Head cross-embodiment policy transformer backbone (NeurIPS 2024 Spotlight) pre-trained across 52 heterogeneous datasets with five ungated Hugging Face checkpoints from 3.1M to 226.8M parameters and an MIT-licensed PyTorch codebase. [1] [2] [3] [4]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Cross-embodiment robot learning researchers; Visuomotor policy practitioners fine-tuning on custom robot datasets; Scaling-law researchers in robot policy learning [1] [3]
- Parameters
- 3.1M (HPT-Small), 12.6M (HPT-Base), 50.5M (HPT-Large), 50.6M (HPT-Base-Lang), 226.8M (HPT-XLarge) [1]
- Embodiments
- Multi-embodiment heterogeneous pretraining across 52 datasets (real-world robots, simulators, deployed robots, human… [1] [3]
- Source rights
- Some licenses are unresolved. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- Ecosystem
- MuJoCo
- License
- MIT · license not fully assessed for model weights, datasets · Details by part
Overview
Heterogeneous Pre-trained Transformers (HPT) is a NeurIPS 2024 Spotlight policy-learning architecture from MIT that pre-trains a shareable transformer trunk across heterogeneous robot embodiments and tasks. Rather than shipping a single monolithic end-to-end VLA, liruiw/HPT provides a modular Stem-Trunk-Head framework and five ungated Hugging Face checkpoints ranging from 3.1M to 226.8M parameters so practitioners can initialize the middle of a policy instead of training from scratch. [1] [3]
Model card summary
- Architecture
Modular Stem-Trunk-Head policy transformer: embodiment-specific perception stems align heterogeneous RGB vision and proprioception (plus language in hpt-base-lang) into a short sequence of latent tokens; a shared pre-trained Transformer trunk processes those tokens; and task/embodiment-specific action heads map trunk representations to robot control actions.
- Parameter count
3.1M (HPT-Small), 12.6M (HPT-Base), 50.5M (HPT-Large), 50.6M (HPT-Base-Lang), 226.8M (HPT-XLarge)
Exact published checkpoint sizes from liruiw/HPT README table.
- Input modalities
- rgbproprioceptionlanguage
Proprioception and visual inputs across all models; language conditioning supported in HPT-Base (With Language).
- Output modalities
- joint-positionsee-pose-deltagripper-command
Modular action heads adapt to embodiment-specific continuous robot action spaces across simulation (Meta-World, MuJoCo) and real-world arms.
- Action space
Embodiment- and task-specific continuous action heads attached to the shared HPT transformer trunk (configurable per dataset/environment via modular YAML configs).
- Trained / supported embodiments
- Multi-embodiment heterogeneous pretraining across 52 datasets (real-world robots, simulators, deployed robots, human video)
- Meta-World / MuJoCo simulation benchmarks
- Custom real-robot setups via realrobot_image.yaml
- Training datasets
- 52 heterogeneous datasets combining large-scale multi-embodiment real-world robot datasets, simulation benchmarks, deployed robot data, and human video datasets (per arXiv:2409.20537v1)
- Interfaces
- Python module CLI (python -m hpt.run)
- Hugging Face Hub URI loader (hf://liruiw/hpt-base)
- Modular YAML configs for stems, trunk, and heads
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| liruiw/hpt-xlarge (226.8M) | pytorch | 1.67 GB | open | 0adad50dbc47 |
| liruiw/hpt-large (50.5M) | pytorch | 0.62 GB | open | e0d9a3711212 |
| liruiw/hpt-base (12.6M) | safetensors | 0.27 GB | open | 4586c8c3918b |
| liruiw/hpt-small (3.1M) | pytorch | 0.09 GB | open | fc12980e7655 |
| liruiw/hpt-base-lang (50.6M; HF API resolves repo ID to liruiw/hpt-large-lang) | pytorch | 0.38 GB | open | 068272250238 |
Each HF repo includes both full model.pth (decimal GB sizes listed) and trunk.pth (0.01–0.91 GB); liruiw/hpt-base also ships model.safetensors (0.05 GB). Note: querying HF for liruiw/hpt-base-lang resolves to repo ID liruiw/hpt-large-lang.
Is this for you?
- Use cases
- Fine-tuning a pre-trained HPT trunk on custom single- or multi-task robot datasets via custom stems and heads
- Benchmarking cross-embodiment transfer on Meta-World and MuJoCo environments
- Studying policy scaling behavior from 3.1M to 226.8M parameters
- Known limitations
- Pre-trained trunk checkpoints are designed for downstream transfer learning (replacing or adapting perception stems and action heads) rather than out-of-the-box zero-shot control on arbitrary robots.
- None of the five Hugging Face checkpoint repositories declare an explicit license YAML tag on the Hub (only the GitHub code repository carries a root MIT LICENSE).
- Custom datasets require implementing convert_dataset, rollout_runner.py, and learner_trajectory_generator.
- Safety and limitations
- Pre-trained trunk weights require fine-tuning with task/embodiment-specific stems and heads before deployment.
- Hugging Face model cards omit explicit license YAML tags, leaving checkpoint license terms formally undeclared on the Hub despite the MIT-licensed GitHub repo.
- Pretraining spans 52 heterogeneous datasets with distinct upstream data licenses.
Compute & fine-tuning
- Inference compute
- Documented hardware: CUDA GPU (exact minimum VRAM not stated in README)
- Precision:
fp32 - Edge / consumer capable: Yes
HPT-Small (3.1M, 0.01 GB trunk.pth) and HPT-Base (12.6M, 0.05 GB safetensors) are lightweight enough for edge deployment, though exact VRAM and latency_ms are not specified.
- Fine-tuning
- Supported: Yes
- Methods: Transfer learning from pre-trained HPT trunk with modular perception stems and action heads (train.pretrained_dir=hf://liruiw/hpt-base)
- Recipe: Upstream fine-tuning documentation
- Host computer
GPU with CUDA recommended for training and fine-tuning; README notes pre-training takes substantial compute and recommends starting from published HF checkpoints.
- Operating system
- Linux
Linux-targeted MuJoCo 2.1.0 EGL and NVIDIA CUDA setup documented in README.
[1] · Editorial assessment
- Dependencies
- Python >= 3.8
- PyTorch >= 2.0
- Hydra / YAML configs
- WandB
- MuJoCo 2.1.0 (optional, for simulation benchmarks)
- Skills
- Python
- PyTorch >= 2.0
- Hydra configuration
- WandB
Inferred from CLI flags (+mode=debug, dataset.use_disk=True) and pyproject.toml setup.
[1] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [1]
- Clone liruiw/HPT and install with pip install -e . (Python >= 3.8, PyTorch >= 2.0)
- Optionally install MuJoCo 2.1.0 for MuJoCo/Meta-World simulation benchmarks
- Review quickstart.ipynb and set train.pretrained_dir to a Hugging Face checkpoint such as hf://liruiw/hpt-base or hf://liruiw/hpt-xlarge
- Run python -m hpt.run or experiments/scripts/metaworld/train_test_metaworld_20task_finetune.sh
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| Multi-simulator and real-world unseen-task transfer (arXiv:2409.20537v1) | >20% improvement in fine-tuned policy performance on unseen tasks across multiple simulator benchmarks and real-world settings | Upstream-reported NeurIPS 2024 Spotlight result after heterogeneous pre-training across 52 datasets; not independently verified by OSR. |
Project notes
Modular Stem-Trunk-Head architecture and five model scales
HPT aligns embodiment-specific vision and proprioception inputs (plus language in the language-conditioned variant) through modular perception stems into a short sequence of latent tokens, processes them with a shared pre-trained Transformer trunk, and maps outputs through task-specific action heads. Five checkpoints are listed in the README: HPT-Small (3.1M, ~0.09 GB / 0.09 GiB model.pth), HPT-Base (12.6M, ~0.27 GB / 0.25 GiB model.pth plus ~0.05 GB model.safetensors), HPT-Large (50.5M, ~0.62 GB / 0.58 GiB), HPT-Base (With Language) (50.6M, ~0.38 GB / 0.35 GiB), and HPT-XLarge (226.8M, ~1.67 GB / 1.56 GiB). Note a naming discrepancy on Hugging Face: the GitHub README links liruiw/hpt-base-lang, whereas Hugging Face API metadata for that URL resolves to repository ID liruiw/hpt-large-lang (068272250238). Each Hub repository includes both full model.pth and standalone trunk.pth weights, loadable via train.pretrained_dir=hf://liruiw/hpt-base. [1] [4] [5]
Fine-tuning workflow and upstream-reported transfer gains
The README documents installation via pip install -e . and lists Python >= 3.8 and PyTorch >= 2.0 (with optional MuJoCo 2.1.0 for Meta-World and MuJoCo benchmarks), alongside an alternative implementation in a LeRobot fork (liruiw/lerobot). Adapting HPT to a custom robot dataset requires implementing a convert_dataset function for training data packing, plus a rollout_runner.py file and a learner_trajectory_generator function for evaluation; perception stem and action head YAML configs (such as realrobot_image.yaml) can be modified as needed. Minimum VRAM and inference latency are not specified in the supplied materials and should not be inferred from checkpoint file sizes. In arXiv:2409.20537v1, the authors report pre-training across 52 heterogeneous datasets (real-world multi-embodiment robot data, simulation, deployed robots, and human video) and achieving over 20% improvement in fine-tuned policy performance on unseen tasks across simulator benchmarks and real-world evaluations. All reported gains are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [3] [4]
What we checked
None. Prepared from public upstream sources without hands-on training, inference, or robot execution.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Upstream last updated
- Dec 6, 2024 [1]
- Latest release
- Oct 2, 2024 [4]
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- liruiw/HPT@549433f70569917eab84dfe0eae9d84a7c4ad9a3; HF liruiw/hpt-base@4586c8c3918b4b6c9f70de925bf4993568f7d8f4 (and hpt-small/large/xlarge/base-lang) · combined hardware and software repository
Separate license matrix
MIT-licensed codebase and five ungated Hugging Face checkpoints (3.1M–226.8M), classified as partial because the Hugging Face weight repositories omit explicit license metadata and pretraining uses 52 mixed-license external datasets.
| Layer | Availability / license |
|---|---|
| Code | available · MIT [1] [2] |
| Model weights | available · License not fully assessed All five liruiw/hpt-* Hugging Face repos are ungated and publicly downloadable, but none declare a license field in their HF model card YAML or include a LICENSE file. [4] [5] |
| Training datasets | available · License not fully assessed Pretrained on a mixture of 52 heterogeneous public/research robot, simulation, and human video datasets with individual upstream licenses. [3] |
| Documentation | available · MIT [1] [2] |
Licensing gap on Hub checkpoints
The liruiw/HPT GitHub repository carries a root MIT License (LICENSE at 549433f70569). However, none of the five liruiw/hpt-* Hugging Face repositories declare a license: tag in their model card YAML or include a LICENSE file, and pretraining draws on 52 external datasets with distinct upstream terms. Commercial users should verify checkpoint and dataset rights directly before redistribution. [1] [2] [3] [4] [5]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 liruiw/HPT GitHub repository and README.md
Documents pip install -e ., Python >= 3.8 / PyTorch >= 2.0, 5 HF checkpoints (3.1M to 226.8M params), Meta-World/custom dataset fine-tuning, and LeRobot fork.Revision, date & where cited
Lirui Wang (GitHub) · github.com · revision
549433f· README.md at commit 549433f70569 (2024-12-06) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 6 more - S002 liruiw/HPT LICENSE
MIT License.Revision, date & where cited
Lirui Wang (GitHub) · github.com · revision
549433f· LICENSE at commit 549433f70569 · Accessed Nov 21, 2026
Used for: Separate license matrix - S003 Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers (NeurIPS 2024 Spotlight)
Describes Stem-Trunk-Head cross-embodiment architecture pre-trained across 52 heterogeneous datasets with >20% fine-tuned improvement on unseen tasks.Revision, date & where cited
arXiv (2409.20537v1) · arxiv.org · arXiv abstract 2409.20537v1 · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Upstream-reported benchmarks and 2 more - S004 Hugging Face model repository: liruiw/hpt-base
Ungated checkpoint with model.pth (0.25 GB), trunk.pth (0.05 GB), and model.safetensors (0.05 GB); no license tag in YAML metadata.Revision, date & where cited
Lirui Wang (Hugging Face) · huggingface.co · HF API & README.md at revision 4586c8c3918b (2024-10-02) · Accessed Nov 21, 2026
Used for: Checkpoints, Is this for you?, Compute & fine-tuning, Modular Stem-Trunk-Head architecture and five model scales and 3 more - S005 Hugging Face model repositories: liruiw/hpt-xlarge, hpt-large, hpt-small, hpt-base-lang
Verified ungated model.pth and trunk.pth sizes across all four sibling repos; none declare an explicit HF license tag.Revision, date & where cited
Lirui Wang (Hugging Face) · huggingface.co · HF API metadata (revisions 0adad50d, e0d9a371, fc12980e, 06827225; 2024-09-27) · Accessed Nov 21, 2026
Used for: Checkpoints, Is this for you?, Modular Stem-Trunk-Head architecture and five model scales, Separate license matrix
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile drafted from liruiw/HPT@549433f70569 and HF liruiw/hpt-* checkpoints.