H2O & OmniH2O (Human-to-Humanoid Whole-Body Control)
Robotics modelCC-BY-NC-4.0 (non-commercial) Isaac Gym sim-to-real RL and DAgger distillation codebase for Unitree H1 whole-body humanoid teleoperation and autonomy (H2O and OmniH2O); upstream releases motion retargeting scripts, filtered AMASS motion data, and hardware wrappers, but not pretrained humanoid policy checkpoints. [1] [2] [3] [4] [5]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Humanoid whole-body control researchers; Sim-to-real RL engineers; Humanoid teleoperation system developers [1] [5]
- Parameters
- Config-dependent compact MLP / RNN actor-critic (not published as a single scalar [1]
- Embodiments
- Unitree H1 / H1 EDU humanoid (with optional Inspire Hands and DM-J4310-2EC wrists) [1]
- Source rights
- Some licenses are unresolved. Includes noncommercial terms. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- Ecosystem
- NVIDIA Jetson
- License
- CC-BY-NC-4.0 (noncommercial) · license not fully assessed for datasets · Details by part
Overview
H2O (IROS 2024) and OmniH2O (CoRL 2024) form CMU LeCAR Lab's whole-body humanoid teleoperation and reinforcement learning control stack for the Unitree H1. Distributed in LeCAR-Lab/human2humanoid, the repository combines AMASS-to-H1 motion retargeting (via PHC), massively parallel Isaac Gym PPO training, DAgger teacher-student policy distillation, and real-robot hardware interface snippets. [1] [4] [5]
Model card summary
- Architecture
Two-stage RL and DAgger distillation architecture in RSL-RL/Legged Gym: a privileged PPO teacher policy (num_observations: 913, num_privileged_obs: 990) distilled into history-conditioned MLP students (0, 5, 25, or 50 history steps; default 25-step student has num_observations: 1665) or recurrent ActorCriticRecurrent (LSTM/GRU) students supporting 3-point VR, 8-point, or 23-point keypoint tracking; also supports direct RL H2O 8-point tracking (num_observations: 138).
- Parameter count
Config-dependent compact MLP / RNN actor-critic (not published as a single scalar; varies across teacher, 0/5/25/50-step MLP students, and LSTM/GRU variants)
Inferred from README Hydra commands showing varying observation dimensions (90 to 3240) and MLP vs ActorCriticRecurrent LSTM/GRU architectures without a single published scalar.
[1] · Editorial assessment
- Input modalities
- proprioceptionimu
Policy takes proprioceptive joint/IMU state history plus retargeted kinematic keypoint targets (VR, RGB pose estimation, or odometry are handled externally).
- Output modalities
- joint-positions
PD target joint positions (control_type: 'P', action_scale: 0.25) for Unitree H1 body joints.
- Action space
Continuous PD joint position targets (control_type: 'P', action_scale: 0.25) for Unitree H1 whole-body joints (hip, knee, ankle, torso, shoulder, elbow); Inspire hands and Damiao wrists are controlled via separate hardware wrappers.
- Control frequency
50 Hz (sim.dt: 0.005 s, control.decimation: 4)
- Trained / supported embodiments
- Unitree H1 / H1 EDU humanoid (with optional Inspire Hands and DM-J4310-2EC wrists)
- Training datasets
- AMASS motion capture dataset (SMPL+H G format, retargeted via PHC)
- Filtered retargeted Unitree H1 motion dataset (amass_phc_filtered.pkl, CMU Box)
Raw AMASS and SMPL assets must be downloaded separately from MPI under their respective licenses.
- Interfaces
- Hydra CLI training and playback scripts (legged_gym/scripts/train_hydra.py, play_hydra.py)
- Unitree SDK2, Inspire Hand C++, Damiao wrist C++, and ZED mini odometry Python wrappers
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| H2O / OmniH2O humanoid policy checkpoints (not released upstream; train teacher and student in Isaac Gym) | pytorch | Not stated | weights-not-released | Not pinned |
Repository only contains inherited legged_gym anydrive_v3_lstm.pt actuator weights, not trained H2O or OmniH2O humanoid policies.
Is this for you?
- Use cases
- Real-time whole-body humanoid teleoperation via VR headset, RGB camera, or sparse keypoints
- Privileged teacher-to-student DAgger distillation for sparse-observation humanoid tracking
- AMASS human motion retargeting and feasibility filtering for Unitree H1
- Known limitations
- Non-commercial license: root repository is licensed under CC-BY-NC-4.0, prohibiting commercial use (e.g., commercial product demos).
- No pretrained H2O or OmniH2O humanoid policy checkpoints are released; users must train teacher and student policies in Isaac Gym.
- Depends on proprietary Isaac Gym Preview 4, Python 3.8, and numpy < 1.24; full retargeting additionally requires separate registration for AMASS and SMPL.
- Hardware deployment code is tailored to the authors' Unitree H1 EDU + ZED mini + Orin NX + Inspire Hand + Damiao wrist rig and relies on external repos for VR/RGB pose estimation.
- Safety and limitations
- CC-BY-NC-4.0 license prohibits commercial use.
- No pretrained humanoid checkpoints are distributed; sim-to-real transfer requires training from scratch in Isaac Gym and careful gain/safety validation on physical humanoids.
- Hardware wrappers are specific to the authors' Unitree H1 EDU + Inspire Hand + Damiao wrist configuration and are not supported for tuning on other robots.
Compute & fine-tuning
- Inference compute
- Documented hardware: NVIDIA Jetson Orin NX (16 GB) onboard Unitree H1 EDU
- Precision:
fp32 - Edge / consumer capable: Yes
README documents real-world deployment on an onboard Jetson Orin NX (16 GB).
- Fine-tuning
- Supported: Yes
- Methods: Privileged PPO teacher RL training in Isaac Gym followed by DAgger student distillation (legged_gym/scripts/train_hydra.py)
- Recipe: Upstream fine-tuning documentation
- Host computer
Single CUDA GPU for Isaac Gym training (default num_envs=4096, sim_device=cuda:0); real-world deployment runs on NVIDIA Jetson Orin NX (16 GB) onboard Unitree H1 EDU.
- Operating system
- Linux
Inferred from Isaac Gym Preview 4 Linux bindings (gym_38.so) and Jetson Orin NX deployment.
[1] · Editorial assessment
- Dependencies
- Python 3.8
- PyTorch
- NVIDIA Isaac Gym Preview 4
- numpy==1.20 (< 1.24)
- setuptools==59.5.0
- Hydra / TensorBoard / WandB
- Bundled rsl_rl, legged_gym, and phc
- Skills
- Python
- PyTorch / RSL-RL
- Isaac Gym / Legged Gym
- Unitree SDK2 / C++ (for hardware deployment)
Inferred from README installation, Hydra RL commands, and C++ Inspire Hand / Damiao wrist hardware files.
[1] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [1]
- Create a Python 3.8 conda environment and install PyTorch and NVIDIA Isaac Gym Preview 4
- Install bundled rsl_rl, legged_gym, phc in editable mode, plus numpy==1.20 (<1.24), setuptools==59.5.0, and requirements.txt
- Download filtered motion data (amass_phc_filtered.pkl from CMU Box) or retarget AMASS + SMPL via grad_fit_h1_shape.py and grad_fit_h1.py
- Train the privileged teacher policy with legged_gym/scripts/train_hydra.py, then distill the sim-to-real student policy via DAgger (train.distill=True)
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| H2O real-world Unitree H1 whole-body RGB teleoperation (arXiv:2403.04436v1) | Zero-shot sim-to-real teleoperation of dynamic whole-body motions (walking, back jumping, kicking, turning, waving, pushing, boxing) | Upstream-reported qualitative/system evaluation on Unitree H1; not independently verified by OSR. |
| OmniH2O universal teleoperation and OmniH2O-6 autonomy suite (arXiv:2406.08858v1) | Demonstrated real-world whole-body teleoperation (VR, RGB, verbal) and autonomous skill learning across six everyday tasks (OmniH2O-6) | Upstream-reported paper result on Unitree H1 with dexterous hands; not independently verified by OSR. |
Project notes
Privileged teacher and sparse-observation student distillation
At the pinned commit (750f1fa05264), training runs in Isaac Gym (sim.dt: 0.005, control.decimation: 4, yielding a 50 Hz policy loop) with PD joint-position actions (control_type: 'P', action_scale: 0.25) across 4,096 parallel environments. OmniH2O first trains a privileged PPO teacher (num_observations: 913, num_privileged_obs: 990) and then distills it via DAgger (train.distill=True) into sim-to-real student policies supporting configurable history lengths (0, 5, 25, or 50 steps; default 25-step MLP uses num_observations: 1665), recurrent ActorCriticRecurrent (LSTM/GRU) backbones, and 3-point VR, 8-point, or 23-point tracking targets. The repository also preserves the direct RL H2O 8-point tracking recipe (num_observations: 138). Exact parameter counts are not stated as single scalars and vary by history/architecture configuration. [1] [2]
Hardware stack, setup caveats, and upstream-reported results
Real-world deployment is documented for a Unitree H1 EDU with an onboard NVIDIA Jetson Orin NX (16 GB), ZED mini odometry, Inspire dexterous hands, and Damiao DM-J4310-2EC wrist actuators; upstream notes the hardware wrappers are specific to that rig and will not be tuned for other robots. Training requires NVIDIA Isaac Gym Preview 4, Python 3.8, and numpy<1.24, while custom motion retargeting requires registering separately for AMASS and SMPL (a pre-filtered amass_phc_filtered.pkl file is linked on CMU Box). In arXiv:2403.04436v1 and arXiv:2406.08858v1, the authors report zero-shot sim-to-real teleoperation of dynamic motions (walking, kicking, waving, boxing, sports) and autonomous skill learning across the OmniH2O-6 task suite. All reported results are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [4] [5]
What we checked
None. Prepared from public upstream sources without hands-on training, inference, or robot execution.
Updates
- Upstream last updated
- Feb 21, 2025 [1]
- Latest release
- Not established
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- LeCAR-Lab/human2humanoid@750f1fa052641f0fde43669d50cb4e407dabe6c8 · software repository
Separate license matrix
Publicly available whole-body humanoid RL and DAgger codebase with filtered retargeted AMASS motion data, included as partial because the repository uses a non-commercial CC-BY-NC-4.0 license, publishes no pretrained humanoid policy checkpoints, and depends on proprietary Isaac Gym Preview 4.
| Layer | Availability / license |
|---|---|
| Code | available · CC-BY-NC-4.0 Root license is Creative Commons Attribution-NonCommercial 4.0 International (non-commercial, not OSI-approved); inherited legged_gym and rsl_rl directories retain BSD-3-Clause notices from ETH Zurich and NVIDIA. [1] [3] |
| Model weights | unavailable · License not fully assessed No pretrained H2O or OmniH2O humanoid policy checkpoints are published in the repository or linked from README. [1] |
| Training datasets | available · License not fully assessed Filtered retargeted Unitree H1 motion file (amass_phc_filtered.pkl) is linked via CMU Box without a separately declared dataset license; full retargeting requires registering for AMASS and SMPL under their upstream non-commercial licenses. [1] |
| Documentation | available · CC-BY-NC-4.0 [1] [3] |
Licensing and weight availability boundaries
The root repository and documentation are licensed under Creative Commons Attribution-NonCommercial 4.0 International (LICENSE-CC-BY-NC-4.0.md), which explicitly forbids commercial use; bundled legged_gym and rsl_rl retain inherited BSD-3-Clause notices from ETH Zurich and NVIDIA. Upstream does not publish pretrained H2O or OmniH2O humanoid policy checkpoints (only an inherited ANYmal anydrive_v3_lstm.pt actuator network appears in legged_gym), and the linked CMU Box motion file does not declare a separate dataset license. [1] [3]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 LeCAR-Lab/human2humanoid GitHub repository and README.md
Documents Isaac Gym Preview 4 setup, teacher/student PPO + DAgger commands (0/5/25/50-step MLP, LSTM/GRU, 3/8/23-point tracking), AMASS/SMPL retargeting, CMU Box link for amass_phc_filtered.pkl, and Unitree H1 EDU + Orin NX hardware stack.Revision, date & where cited
CMU LeCAR Lab (GitHub) · github.com · revision
750f1fa· README.md at commit 750f1fa05264 (2025-02-21) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 6 more - S002 human2humanoid Hydra control and simulation configs (control_teleop.yaml, sim_teleop.yaml, config_teleop.yaml)
Specifies PD position control (control_type: 'P', action_scale: 0.25, decimation: 4) with sim dt: 0.005 s (50 Hz policy frequency) and num_envs: 4096.Revision, date & where cited
CMU LeCAR Lab (GitHub) · github.com · revision
750f1fa· control_teleop.yaml and sim_teleop.yaml at commit 750f1fa05264 · Accessed Nov 21, 2026
Used for: Model card summary, Compute & fine-tuning, Privileged teacher and sparse-observation student distillation - S003 LeCAR-Lab/human2humanoid LICENSE-CC-BY-NC-4.0.md
Creative Commons Attribution-NonCommercial 4.0 International license; README explicitly notes commercial use is prohibited and legged_gym/rsl_rl retain inherited BSD-3-Clause licenses.Revision, date & where cited
CMU LeCAR Lab (GitHub) · github.com · revision
750f1fa· LICENSE-CC-BY-NC-4.0.md at commit 750f1fa05264 · Accessed Nov 21, 2026
Used for: Is this for you?, Separate license matrix - S004 Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation (IROS 2024 / arXiv:2403.04436v1)
Introduces H2O sim-to-data feasibility filtering and zero-shot sim-to-real RL whole-body teleoperation on a full-sized humanoid with an RGB camera.Revision, date & where cited
arXiv (2403.04436v1) · arxiv.org · arXiv abstract 2403.04436v1 · Accessed Nov 21, 2026
Used for: Overview, Is this for you?, Upstream-reported benchmarks, Hardware stack, setup caveats, and upstream-reported results - S005 OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning (CoRL 2024 / arXiv:2406.08858v1)
Extends H2O to universal VR/RGB/verbal teleoperation and autonomous skill learning via teacher-student distillation and introduces the OmniH2O-6 task suite.Revision, date & where cited
arXiv (2406.08858v1) · arxiv.org · arXiv abstract 2406.08858v1 · Accessed Nov 21, 2026
Used for: Overview, Is this for you?, Upstream-reported benchmarks, Hardware stack, setup caveats, and upstream-reported results
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile drafted from LeCAR-Lab/human2humanoid@750f1fa05264.