Improved 3D Diffusion Policy (iDP3)
Robotics modelMIT-licensed egocentric 3D visuomotor diffusion policy for humanoid manipulation that operates on unsegmented, uncalibrated 4096-point LiDAR/depth point clouds via a multi-stage PointNet encoder and 1D conditional U-Net DDIM action head; upstream releases training/deployment code, datasets, and 3-task Fourier GR1 checkpoints on Google Drive. [1] [2] [3] [4]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Humanoid manipulation researchers; 3D visuomotor policy engineers; Real-world imitation learning practitioners [1] [4]
- Parameters
- Task-dependent (not published as a single scalar [2]
- Embodiments
- Fourier GR1 humanoid (25-DoF upper body + dexterous hands with RealSense L515) [1] [2] [4]
- Source rights
- Some licenses are unresolved. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- License
- MIT · license not fully assessed for model weights, datasets · Details by part
Overview
Improved 3D Diffusion Policy (iDP3) is an MIT-licensed visuomotor diffusion policy accepted at IROS 2025 for real-world humanoid manipulation. The policy operates on egocentric 3D point clouds without camera calibration or point-cloud segmentation. The official repository (YanjieZe/Improved-3D-Diffusion-Policy) distributes training scripts, example deployment scripts, and Google Drive links to datasets and checkpoints for three real-world Fourier GR1 tasks. [1] [3] [4]
Model card summary
- Architecture
Multi-stage PointNet egocentric 3D point-cloud encoder (multi_stage_pointnet, in_channels: 3, out_channels: 128, num_points: 4096, use_pc_color: false, use_layernorm: true) without camera calibration or segmentation, fused with 32-D proprioception (agent_pos) as FiLM global conditioning into a 1D conditional U-Net (down_dims: [256, 512, 1024], kernel_size: 5) with DDIMScheduler (50 training timesteps, 10 inference steps, prediction_type: sample). Also includes a 2D R3M image-policy baseline (dp_224x224_r3m.yaml).
- Parameter count
Task-dependent (not published as a single scalar; uses multi_stage_pointnet with 128-D output + 1D U-Net with down_dims [256, 512, 1024])
Exact parameter count depends on task observation/action dimensions.
[2] · Editorial assessment
- Input modalities
- point-cloudproprioception
4096-point egocentric 3D point clouds from RealSense L515 (stored as [4096, 6] in zarr, xyz used with use_pc_color: false) plus 32-D agent_pos proprioception.
- Output modalities
- joint-positions
25-D action vector for the 25-DoF Fourier GR1 upper-body and dexterous hand platform (gr1_dex-3d.yaml).
- Action space
Continuous DDIM-denoised action chunks (horizon: 16, n_obs_steps: 2, n_action_steps: 15, 10 DDIM inference steps in idp3.yaml); 25-D upper-body + hand action space for Fourier GR1 in gr1_dex-3d.yaml.
- Trained / supported embodiments
- Fourier GR1 humanoid (25-DoF upper body + dexterous hands with RealSense L515)
README notes iDP3 can be adapted to other robots equipped with a 3D depth camera.
- Training datasets
- Fourier GR1 real-world 3-task zarr demonstration dataset (Google Drive)
- Example training dataset (training_data_example, Google Drive)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| iDP3 pretrained checkpoints (all 3 Fourier GR1 real-world tasks, Google Drive folder) | pytorch | Not stated | open | unversioned Google Drive folder (released 2024-11-04, linked from f5b27faabbb8) |
Released 2024-11-04 alongside full 3-task datasets in Google Drive; no separate weights license is declared.
Is this for you?
- Use cases
- Autonomous bimanual and dexterous humanoid manipulation in unseen scenes
- Training 3D visuomotor policies from noisy human teleoperation demonstrations
- Onboard CPU policy deployment on full-sized humanoids equipped with 3D depth/LiDAR sensors
- Known limitations
- Sensitive to depth camera quality: README explicitly warns that Intel RealSense D435 depth data is too imprecise and recommends RealSense L515.
- Deployment script is a hardware example rather than a turnkey package; missing communication modules (e.g., communication.py) live in the companion YanjieZe/Humanoid-Teleoperation repository and require robot-specific API adaptation.
- Environment pins Python 3.8, PyTorch 2.1.0, diffusers 0.11.1, zarr 2.12.0, and pyrealsense2 2.54.2.5684.
- Published Google Drive checkpoints and datasets do not declare a separate license.
- Safety and limitations
- Imprecise depth sensors such as Intel RealSense D435 are explicitly discouraged by upstream for 3D policy training and deployment.
- Deployment scripts are reference examples that depend on external Zenoh communication code (Humanoid-Teleoperation) and robot-specific low-level APIs.
- Trained per task from single-scene demonstrations without built-in hardware collision-avoidance guarantees.
Compute & fine-tuning
- Inference compute
- Documented hardware: Onboard CPU of Fourier GR1 for deployment; NVIDIA RTX 4090 (24 GB) for training
- Precision:
fp32 - Edge / consumer capable: Yes
README and arXiv abstract explicitly report running autonomous policy deployment solely on the onboard CPU of the Fourier GR1.
- Fine-tuning
- Supported: Yes
- Methods: Per-task 3D diffusion behavior cloning on zarr point-cloud datasets (scripts/train_policy.sh idp3)
- Minimum VRAM: 24 GB
- Recipe: Upstream fine-tuning documentation
Training machine reported in README is an NVIDIA RTX 4090 (24 GB); exact lower VRAM bound is not stated.
- Host computer
Trained on a local workstation with an NVIDIA RTX 4090 (24 GB VRAM); deployed on the onboard CPU of the Fourier GR1 humanoid.
- Operating system
- Linux
Inferred from bash scripts, CUDA wheel setup, and Fourier GR1 onboard Linux deployment.
[1] · Editorial assessment
- Dependencies
- Python 3.8
- PyTorch 2.1.0
- diffusers==0.11.1
- hydra-core==1.2.0
- zarr==2.12.0
- open3d / torch_geometric
- pyrealsense2==2.54.2.5684 (deployment)
- Skills
- Python
- PyTorch
- 3D point-cloud processing / RealSense SDK
- Robot teleoperation & Zenoh communication
Inferred from README installation, RealSense L515 driver setup, and companion teleop-zenoh reference.
[1] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [1]
- Create a Python 3.8 conda environment and install PyTorch 2.1.0 matching host CUDA
- Install third_party/visualizer, required diffusion/point-cloud Python packages, and Improved-3D-Diffusion-Policy in editable mode (pip install -e .)
- On deployment machines, install librealsense v2.54.2 and pyrealsense2==2.54.2.5684 for RealSense L515
- Download the example or full dataset and checkpoints from Google Drive, set dataset_path in scripts/train_policy.sh, and run train_policy.sh or deploy_policy.sh
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| Real-world Fourier GR1 autonomous humanoid manipulation across diverse unseen scenes (arXiv:2410.10803v3) | Over 2,000 real-robot policy rollout episodes demonstrating single-scene-to-diverse-scene skill generalization using only onboard CPU compute | Upstream-reported real-world evaluation across 3 tasks on a 25-DoF Fourier GR1 with RealSense L515; not independently verified by OSR. |
Project notes
Multi-stage PointNet encoder and DDIM action head
At the pinned commit (f5b27faabbb8), idp3.yaml pairs a multi_stage_pointnet encoder (in_channels: 3, out_channels: 128, num_points: 4096, use_pc_color: false, use_layernorm: true) with 32-D proprioception (agent_pos in gr1_dex-3d.yaml) as FiLM global conditioning into a 1D conditional U-Net (down_dims: [256, 512, 1024], kernel_size: 5). Action chunks (horizon: 16, n_obs_steps: 2, n_action_steps: 15, 25-D upper-body + hand actions for Fourier GR1) are generated via DDIMScheduler with 50 training timesteps, 10 inference steps, and prediction_type: sample. The repository also includes a 2D R3M image-based Diffusion Policy baseline (dp_224x224_r3m.yaml). Exact parameter counts and a single closed-loop Hz figure are not pinned in the README. [1] [2]
Deployment constraints and upstream-reported evaluation
Upstream trains policies on a workstation with an NVIDIA RTX 4090 (24 GB) and deploys them solely on the onboard CPU of the Fourier GR1 humanoid using an Intel RealSense L515 depth camera (librealsense v2.54.2, pyrealsense2==2.54.2.5684). The README explicitly warns that RealSense D435 depth data is too imprecise for training the 3D policy. Furthermore, scripts/deploy_policy.sh is an example rather than a standalone deployment bundle: missing communication modules such as communication.py reside in the companion YanjieZe/Humanoid-Teleoperation repository and must be adapted to target robot APIs. In arXiv:2410.10803v3, the authors report over 2,000 real-robot rollout episodes on a 25-DoF humanoid demonstrating autonomous generalization to diverse real-world scenes from single-scene training data using onboard compute. All reported metrics are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [4]
What we checked
None. Prepared from public upstream sources without hands-on training, inference, or robot execution.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Upstream last updated
- Jun 16, 2025 [1]
- Latest release
- Not established
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- YanjieZe/Improved-3D-Diffusion-Policy@f5b27faabbb8952672dd3d6a21d0ba762d5dfabb · software repository
Separate license matrix
MIT-licensed 3D humanoid manipulation policy with downloadable 3-task checkpoints and datasets on Google Drive, included as partial because the Drive checkpoints and datasets lack an explicit separate license and real-robot deployment depends on external Zenoh communication code.
| Layer | Availability / license |
|---|---|
| Code | available · MIT Root LICENSE is MIT (Copyright 2024 Yanjie Ze); third_party/r3m retains its upstream license. [1] [3] |
| Model weights | available · License not fully assessed Checkpoints for all 3 real-world tasks are hosted in a Google Drive folder without a separately declared weights license. [1] |
| Training datasets | available · License not fully assessed Full 3-task dataset and example zarr dataset are hosted on Google Drive without a separately declared dataset license. [1] |
| Documentation | available · MIT [1] [3] |
Licensing and weight availability boundaries
The codebase and documentation are released under the MIT License (LICENSE at f5b27faabbb8, Copyright 2024 Yanjie Ze), while bundled third_party/r3m retains its upstream license. Checkpoints and full datasets for all three tasks are downloadable from Google Drive, but neither the weights nor the datasets declare a separate license. [1] [3]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 YanjieZe/Improved-3D-Diffusion-Policy GitHub repository and README.md
Documents IROS 2025 acceptance, RTX 4090 training and Fourier GR1 onboard CPU deployment, RealSense L515 requirement (warning against D435), Python 3.8 / PyTorch 2.1.0 setup, and Google Drive links to example data and full 3-task data/checkpoints.Revision, date & where cited
Yanjie Ze (GitHub) · github.com · revision
f5b27fa· README.md at commit f5b27faabbb8 (2025-06-16) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 7 more - S002 iDP3 Hydra configurations (idp3.yaml and gr1_dex-3d.yaml)
Specifies multi_stage_pointnet (in_channels: 3, out_channels: 128, num_points: 4096, use_pc_color: false), 1D U-Net (down_dims: [256, 512, 1024]), DDIMScheduler (50 train steps, 10 inference steps, prediction_type: sample), horizon: 16, n_obs_steps: 2, n_action_steps: 15, 32-D agent_pos, and 25-D action.Revision, date & where cited
Yanjie Ze (GitHub) · github.com · revision
f5b27fa· idp3.yaml and task/gr1_dex-3d.yaml at commit f5b27faabbb8 · Accessed Nov 21, 2026
Used for: Model card summary, Multi-stage PointNet encoder and DDIM action head - S003 YanjieZe/Improved-3D-Diffusion-Policy LICENSE
MIT License, Copyright (c) 2024 Yanjie Ze.Revision, date & where cited
Yanjie Ze (GitHub) · github.com · revision
f5b27fa· LICENSE at commit f5b27faabbb8 · Accessed Nov 21, 2026
Used for: Overview, Separate license matrix - S004 Generalizable Humanoid Manipulation with 3D Diffusion Policies (IROS 2025 / arXiv:2410.10803v3)
Describes 25-DoF humanoid platform with 3D LiDAR and iDP3 policy evaluated across >2,000 real-robot rollout episodes using only single-scene training data and onboard compute.Revision, date & where cited
arXiv (2410.10803v3) · arxiv.org · arXiv abstract 2410.10803v3 · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Compute & fine-tuning and 2 more
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile drafted from YanjieZe/Improved-3D-Diffusion-Policy@f5b27faabbb8.