Open Source RoboticsRobots, software & learning

Improved 3D Diffusion Policy (iDP3)

Robotics model

MIT-licensed egocentric 3D visuomotor diffusion policy for humanoid manipulation that operates on unsegmented, uncalibrated 4096-point LiDAR/depth point clouds via a multi-stage PointNet encoder and 1D conditional U-Net DDIM action head; upstream releases training/deployment code, datasets, and 3-task Fourier GR1 checkpoints on Google Drive. [1] [2] [3] [4]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Humanoid manipulation researchers; 3D visuomotor policy engineers; Real-world imitation learning practitioners [1] [4]
Parameters
Task-dependent (not published as a single scalar [2]
Embodiments
Fourier GR1 humanoid (25-DoF upper body + dexterous hands with RealSense L515) [1] [2] [4]
Source rights
Some licenses are unresolved. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware
License
MIT · license not fully assessed for model weights, datasets · Details by part

Requirements · Research details and dates

On this page

Overview

Improved 3D Diffusion Policy (iDP3) is an MIT-licensed visuomotor diffusion policy accepted at IROS 2025 for real-world humanoid manipulation. The policy operates on egocentric 3D point clouds without camera calibration or point-cloud segmentation. The official repository (YanjieZe/Improved-3D-Diffusion-Policy) distributes training scripts, example deployment scripts, and Google Drive links to datasets and checkpoints for three real-world Fourier GR1 tasks. [1] [3] [4]

Model card summary

Architecture

Multi-stage PointNet egocentric 3D point-cloud encoder (multi_stage_pointnet, in_channels: 3, out_channels: 128, num_points: 4096, use_pc_color: false, use_layernorm: true) without camera calibration or segmentation, fused with 32-D proprioception (agent_pos) as FiLM global conditioning into a 1D conditional U-Net (down_dims: [256, 512, 1024], kernel_size: 5) with DDIMScheduler (50 training timesteps, 10 inference steps, prediction_type: sample). Also includes a 2D R3M image-policy baseline (dp_224x224_r3m.yaml).

[1] [2]

Parameter count

Task-dependent (not published as a single scalar; uses multi_stage_pointnet with 128-D output + 1D U-Net with down_dims [256, 512, 1024])

Exact parameter count depends on task observation/action dimensions.

[2] · Editorial assessment

Input modalities
point-cloudproprioception

4096-point egocentric 3D point clouds from RealSense L515 (stored as [4096, 6] in zarr, xyz used with use_pc_color: false) plus 32-D agent_pos proprioception.

[1] [2]

Output modalities
joint-positions

25-D action vector for the 25-DoF Fourier GR1 upper-body and dexterous hand platform (gr1_dex-3d.yaml).

[2] [4]

Action space

Continuous DDIM-denoised action chunks (horizon: 16, n_obs_steps: 2, n_action_steps: 15, 10 DDIM inference steps in idp3.yaml); 25-D upper-body + hand action space for Fourier GR1 in gr1_dex-3d.yaml.

[2] [4]

Trained / supported embodiments
  • Fourier GR1 humanoid (25-DoF upper body + dexterous hands with RealSense L515)

README notes iDP3 can be adapted to other robots equipped with a 3D depth camera.

[1] [2] [4]

Training datasets
  • Fourier GR1 real-world 3-task zarr demonstration dataset (Google Drive)
  • Example training dataset (training_data_example, Google Drive)

[1]

Interfaces
  • Hydra YAML CLI scripts (scripts/train_policy.sh, scripts/deploy_policy.sh, scripts/vis_dataset.sh)
  • Zarr point-cloud demonstration datasets
  • Intel RealSense L515 wrapper and Zenoh robot communication (via companion Humanoid-Teleoperation repo)

[1] [2]

Checkpoints

CheckpointFormatSizeAccessPinned revision
iDP3 pretrained checkpoints (all 3 Fourier GR1 real-world tasks, Google Drive folder)pytorchNot statedopenunversioned Google Drive folder (released 2024-11-04, linked from f5b27faabbb8)

Released 2024-11-04 alongside full 3-task datasets in Google Drive; no separate weights license is declared.

[1]

Is this for you?

Use cases
  • Autonomous bimanual and dexterous humanoid manipulation in unseen scenes
  • Training 3D visuomotor policies from noisy human teleoperation demonstrations
  • Onboard CPU policy deployment on full-sized humanoids equipped with 3D depth/LiDAR sensors

[1] [4]

Known limitations
  • Sensitive to depth camera quality: README explicitly warns that Intel RealSense D435 depth data is too imprecise and recommends RealSense L515.
  • Deployment script is a hardware example rather than a turnkey package; missing communication modules (e.g., communication.py) live in the companion YanjieZe/Humanoid-Teleoperation repository and require robot-specific API adaptation.
  • Environment pins Python 3.8, PyTorch 2.1.0, diffusers 0.11.1, zarr 2.12.0, and pyrealsense2 2.54.2.5684.
  • Published Google Drive checkpoints and datasets do not declare a separate license.

[1]

Safety and limitations
  • Imprecise depth sensors such as Intel RealSense D435 are explicitly discouraged by upstream for 3D policy training and deployment.
  • Deployment scripts are reference examples that depend on external Zenoh communication code (Humanoid-Teleoperation) and robot-specific low-level APIs.
  • Trained per task from single-scene demonstrations without built-in hardware collision-avoidance guarantees.

[1]

Compute & fine-tuning

Inference compute
  • Documented hardware: Onboard CPU of Fourier GR1 for deployment; NVIDIA RTX 4090 (24 GB) for training
  • Precision: fp32
  • Edge / consumer capable: Yes

README and arXiv abstract explicitly report running autonomous policy deployment solely on the onboard CPU of the Fourier GR1.

[1] [4]

Fine-tuning
  • Supported: Yes
  • Methods: Per-task 3D diffusion behavior cloning on zarr point-cloud datasets (scripts/train_policy.sh idp3)
  • Minimum VRAM: 24 GB
  • Recipe: Upstream fine-tuning documentation

Training machine reported in README is an NVIDIA RTX 4090 (24 GB); exact lower VRAM bound is not stated.

[1]

Host computer

Trained on a local workstation with an NVIDIA RTX 4090 (24 GB VRAM); deployed on the onboard CPU of the Fourier GR1 humanoid.

[1]

Operating system
  • Linux

Inferred from bash scripts, CUDA wheel setup, and Fourier GR1 onboard Linux deployment.

[1] · Editorial assessment

Dependencies
  • Python 3.8
  • PyTorch 2.1.0
  • diffusers==0.11.1
  • hydra-core==1.2.0
  • zarr==2.12.0
  • open3d / torch_geometric
  • pyrealsense2==2.54.2.5684 (deployment)

[1]

Skills
  • Python
  • PyTorch
  • 3D point-cloud processing / RealSense SDK
  • Robot teleoperation & Zenoh communication

Inferred from README installation, RealSense L515 driver setup, and companion teleop-zenoh reference.

[1] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1] · Model card / documentation [1]

  • Create a Python 3.8 conda environment and install PyTorch 2.1.0 matching host CUDA
  • Install third_party/visualizer, required diffusion/point-cloud Python packages, and Improved-3D-Diffusion-Policy in editable mode (pip install -e .)
  • On deployment machines, install librealsense v2.54.2 and pyrealsense2==2.54.2.5684 for RealSense L515
  • Download the example or full dataset and checkpoints from Google Drive, set dataset_path in scripts/train_policy.sh, and run train_policy.sh or deploy_policy.sh

[1]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
Real-world Fourier GR1 autonomous humanoid manipulation across diverse unseen scenes (arXiv:2410.10803v3)Over 2,000 real-robot policy rollout episodes demonstrating single-scene-to-diverse-scene skill generalization using only onboard CPU computeUpstream-reported real-world evaluation across 3 tasks on a 25-DoF Fourier GR1 with RealSense L515; not independently verified by OSR.

[1] [4]

Project notes

Multi-stage PointNet encoder and DDIM action head

At the pinned commit (f5b27faabbb8), idp3.yaml pairs a multi_stage_pointnet encoder (in_channels: 3, out_channels: 128, num_points: 4096, use_pc_color: false, use_layernorm: true) with 32-D proprioception (agent_pos in gr1_dex-3d.yaml) as FiLM global conditioning into a 1D conditional U-Net (down_dims: [256, 512, 1024], kernel_size: 5). Action chunks (horizon: 16, n_obs_steps: 2, n_action_steps: 15, 25-D upper-body + hand actions for Fourier GR1) are generated via DDIMScheduler with 50 training timesteps, 10 inference steps, and prediction_type: sample. The repository also includes a 2D R3M image-based Diffusion Policy baseline (dp_224x224_r3m.yaml). Exact parameter counts and a single closed-loop Hz figure are not pinned in the README. [1] [2]

Deployment constraints and upstream-reported evaluation

Upstream trains policies on a workstation with an NVIDIA RTX 4090 (24 GB) and deploys them solely on the onboard CPU of the Fourier GR1 humanoid using an Intel RealSense L515 depth camera (librealsense v2.54.2, pyrealsense2==2.54.2.5684). The README explicitly warns that RealSense D435 depth data is too imprecise for training the 3D policy. Furthermore, scripts/deploy_policy.sh is an example rather than a standalone deployment bundle: missing communication modules such as communication.py reside in the companion YanjieZe/Humanoid-Teleoperation repository and must be adapted to target robot APIs. In arXiv:2410.10803v3, the authors report over 2,000 real-robot rollout episodes on a 25-DoF humanoid demonstrating autonomous generalization to diverse real-world scenes from single-scene training data using onboard compute. All reported metrics are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [4]

What we checked

None. Prepared from public upstream sources without hands-on training, inference, or robot execution.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Jun 16, 2025 [1]
Latest release
Not established
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
YanjieZe/Improved-3D-Diffusion-Policy@f5b27faabbb8952672dd3d6a21d0ba762d5dfabb · software repository

Separate license matrix

MIT-licensed 3D humanoid manipulation policy with downloadable 3-task checkpoints and datasets on Google Drive, included as partial because the Drive checkpoints and datasets lack an explicit separate license and real-robot deployment depends on external Zenoh communication code.

LayerAvailability / license
Codeavailable · MIT
Root LICENSE is MIT (Copyright 2024 Yanjie Ze); third_party/r3m retains its upstream license. [1] [3]
Model weightsavailable · License not fully assessed
Checkpoints for all 3 real-world tasks are hosted in a Google Drive folder without a separately declared weights license. [1]
Training datasetsavailable · License not fully assessed
Full 3-task dataset and example zarr dataset are hosted on Google Drive without a separately declared dataset license. [1]
Documentationavailable · MIT
[1] [3]

Licensing and weight availability boundaries

The codebase and documentation are released under the MIT License (LICENSE at f5b27faabbb8, Copyright 2024 Yanjie Ze), while bundled third_party/r3m retains its upstream license. Checkpoints and full datasets for all three tasks are downloadable from Google Drive, but neither the weights nor the datasets declare a separate license. [1] [3]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile drafted from YanjieZe/Improved-3D-Diffusion-Policy@f5b27faabbb8.