3D Diffusion Policy (DP3)
Robotics modelMIT-licensed 3D visuomotor diffusion policy architecture combining a compact PointNet-style sparse point-cloud encoder with a 1D conditional U-Net DDIM action head for data-efficient dexterous and arm manipulation; upstream releases training/evaluation code, RL demonstration experts, and real-robot datasets, but not pretrained DP3 policy weights. [1] [2] [3] [4]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Robot learning researchers; Dexterous manipulation practitioners; Imitation learning engineers using depth cameras [1] [5]
- Embodiments
- Franka Emika Panda + Allegro Hand (real robot reference), Adroit Shadow Hand (simulation), DexArt articulated-object… [1]
- Source rights
- Some licenses are unresolved. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- Ecosystem
- MuJoCo
- License
- MIT · license not fully assessed for datasets · Details by part
Overview
3D Diffusion Policy (DP3) is an MIT-licensed visuomotor imitation learning architecture from RSS 2024 that pairs a compact 3D point-cloud encoder with a 1D conditional U-Net diffusion policy. Rather than shipping a generalist pretrained checkpoint, the upstream repository (YanjieZe/3D-Diffusion-Policy) provides the reference training and evaluation codebase for task-specific behavior cloning across dexterous hand and arm manipulation tasks, together with simulation benchmark wrappers and real-robot demonstration data. [1] [3] [4]
Model card summary
- Architecture
Compact PointNet-style sparse 3D point-cloud encoder (LayerNorm, encoder_output_dim: 64, use_pc_color: false by default) combined with robot proprioception as FiLM global conditioning into a 1D conditional U-Net DDIM diffusion action head (down_dims: [512, 1024, 2048] in dp3.yaml; 100 training timesteps, 10 DDIM inference steps). Also ships a faster Simple DP3 variant (simple_dp3.yaml).
- Parameter count
Task-dependent (not pinned as a single scalar in README or dp3.yaml; uses compact PointNet encoder with 64-D output + 1D U-Net with down_dims [512, 1024, 2048] in dp3.yaml)
Exact parameter count depends on task state/action dimensions and whether dp3.yaml or simple_dp3.yaml is used.
- Input modalities
- point-cloudproprioception
Sparse 3D point clouds (shape (T, Np, 3) xyz by default with use_pc_color: false, or (T, Np, 6) xyzrgb in real-robot format) plus agent_pos proprioception.
- Output modalities
- ee-pose-deltajoint-positions
Real-robot README documents 22-D action space using relative end-effector position control (6-D) for the Franka arm and relative joint-angle position control (16-D) for the Allegro hand; README also notes global position action space can improve results.
- Action space
Continuous DDIM-denoised action chunks (default horizon: 16, n_obs_steps: 2, n_action_steps: 8, 10 DDIM inference steps in dp3.yaml); task-specific dimensions (e.g., 22-D for Franka 6-D arm + Allegro 16-D hand).
- Control frequency
Up to 25 Hz inference reported for Simple DP3 (25 FPS per README)
Full dp3.yaml inference FPS is not given as a single scalar in the README.
- Trained / supported embodiments
- Franka Emika Panda + Allegro Hand (real robot reference)
- Adroit Shadow Hand (simulation)
- DexArt articulated-object manipulation (simulation)
- MetaWorld Sawyer arm (simulation)
- Training datasets
- Adroit demonstrations generated from provided VRL3 RL expert checkpoints
- DexArt demonstrations generated from provided assets
- MetaWorld demonstrations (50 tasks)
- Real-world Franka + Allegro zarr demonstration dataset (Google Drive)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| DP3 policy checkpoints (not released upstream; train per task from demonstrations) | pytorch | Not stated | weights-not-released | Not pinned |
Upstream releases VRL3 RL expert checkpoints solely for generating Adroit simulation demonstrations, not pretrained DP3 diffusion policy checkpoints.
Is this for you?
- Use cases
- Training task-specific 3D visuomotor policies from 10 to 40 demonstrations
- Benchmarking 3D imitation learning on Adroit, DexArt, and MetaWorld
- Real-world dexterous manipulation with cropped point clouds from depth cameras
- Known limitations
- No pretrained DP3 policy checkpoints are published upstream; users must generate demonstrations or supply custom zarr datasets and train per task.
- Sensitive to depth sensor quality: README warns that Intel RealSense D435 point clouds can cause failure compared to RealSense L515.
- Requires manual point-cloud bounding-box cropping and FPS downsampling to remove table/background points in real-world deployments.
- Simulation installation pins legacy Python 3.8, gym 0.21.0, MuJoCo 2.1.0, and mujoco-py 2.1.2.14.
- Safety and limitations
- No pretrained policy weights are distributed; every target robot/task requires demonstration collection and per-task training.
- Depth sensor noise directly degrades policy behavior; README warns RealSense D435 point clouds can cause failures.
- Real-robot deployment code is not self-contained in this repo; README directs users to Improved-3D-Diffusion-Policy (iDP3) for deployment reference.
Compute & fine-tuning
- Inference compute
- Documented hardware: NVIDIA RTX 3090 / A40 / A800 / A100 (per INSTALL.md and README)
- Precision:
fp32 - Reported latency: 40 ms
40 ms corresponds to the README's 25 FPS inference figure for simple_dp3.yaml; training uses ~10 GB VRAM on an A40, while exact minimum inference VRAM is not separately stated.
- Fine-tuning
- Supported: Yes
- Methods: From-scratch per-task behavior cloning on zarr point-cloud datasets (dp3.yaml / simple_dp3.yaml)
- Minimum VRAM: 10 GB
- Recipe: Upstream fine-tuning documentation
README reports ~10 GB GPU memory and ~3 hours on an A40 for dp3.yaml (1-2 hours for simple_dp3.yaml).
- Host computer
Tested on NVIDIA RTX 3090 / A40 / A800 / A100 with CUDA 11.7+; README reports ~10 GB GPU memory and ~3 hours training on an A40 for dp3.yaml (1-2 hours for simple_dp3.yaml).
- Dependencies
- Python 3.8
- PyTorch
- diffusers==0.11.1
- hydra-core==1.2.0
- zarr==2.12.0
- MuJoCo 2.1.0 / mujoco-py-2.1.2.14
- pytorch3d_simplified (bundled)
Setup & usage
Open the upstream getting-started guide [5] · Model card / documentation [1]
- Create a Python 3.8 conda environment and install PyTorch matching host CUDA
- Install 3D-Diffusion-Policy in editable mode (pip install -e .)
- Install MuJoCo 2.1.0, bundled third_party/mujoco-py-2.1.2.14, gym-0.21.0, DexArt, MetaWorld, and simplified PyTorch3D
- Download VRL3 Adroit RL experts and DexArt assets or real-robot zarr data, generate demonstrations, and run scripts/train_policy.sh
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| 72 simulation tasks (Adroit, DexArt, MetaWorld, etc. in arXiv:2403.03954v7) | 24.2% relative improvement over baselines with 10 demonstrations | Upstream-reported paper result; repository bundles 57 tasks (3 Adroit + 4 DexArt + 50 MetaWorld) and notes regenerated demonstrations can vary from paper numbers. Not independently verified by OSR. |
| 4 real-robot manipulation tasks (Franka + Allegro Hand + RealSense L515, arXiv:2403.03954v7) | 85% average success rate with 40 demonstrations per task | Upstream-reported real-robot evaluation using cropped L515 point clouds; not independently verified by OSR. |
Project notes
Compact 3D encoder and DDIM action head
At the pinned commit (47385d9d6f5b), dp3.yaml configures a lightweight PointNet-style encoder (in_channels: 3, out_channels: 64, use_layernorm: true, use_pc_color: false) that compresses sparse 3D point clouds and fuses them with robot proprioception (agent_pos) as FiLM global conditioning into a 1D U-Net (down_dims: [512, 1024, 2048]). Actions are generated with a DDIMScheduler using 100 training timesteps and 10 inference denoising steps (horizon: 16, n_obs_steps: 2, n_action_steps: 8). The repository also provides simple_dp3.yaml, a lighter variant recommended by the authors for faster training (1–2 hours) and inference (up to 25 FPS). Exact parameter counts are not stated as a single scalar in the README or config and depend on task state/action dimensionality and config variant. [1] [2]
Setup constraints and upstream-reported benchmarks
The repository bundles 57 simulation tasks across Adroit (3), DexArt (4), and MetaWorld (50), plus a real-world Franka Emika Panda + Allegro Hand + Intel RealSense L515 demonstration dataset (zarr). Training dp3.yaml requires roughly 10 GB of GPU memory and ~3 hours on an NVIDIA A40, while installation pins legacy simulation dependencies including Python 3.8, gym==0.21.0, MuJoCo 2.1.0, and mujoco-py==2.1.2.14. Upstream warns that lower-quality depth cameras such as the RealSense D435 can cause DP3 failures and recommends cropping table/background points before FPS downsampling. Note also that real-world robot deployment code is not self-contained in this repository; upstream directs readers to Improved-3D-Diffusion-Policy (iDP3) as a deployment reference, so this codebase and dataset should not be treated as a complete hardware deployment package. In arXiv:2403.03954v7, the authors report a 24.2% relative improvement over baselines across 72 simulation tasks using 10 demonstrations and an 85% average success rate across 4 real-robot tasks using 40 demonstrations each. All reported metrics are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [4] [5]
What we checked
None. Prepared from public upstream sources without hands-on training, inference, or robot execution.
Updates
- Upstream last updated
- Oct 17, 2025 [1]
- Latest release
- Not established
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- YanjieZe/3D-Diffusion-Policy@47385d9d6f5bde3f2ebdf2400ecb8261cc9e6b97 · software repository
Separate license matrix
MIT-licensed 3D visuomotor diffusion policy codebase with bundled simulation benchmarks and real-robot zarr datasets, included as partial because upstream does not publish pretrained DP3 policy checkpoints and linked Drive datasets lack an explicit separate license.
| Layer | Availability / license |
|---|---|
| Code | available · MIT Root LICENSE is MIT (Copyright 2024 Yanjie Ze); third_party/ bundles external simulation dependencies with their own upstream licenses. [1] [3] |
| Model weights | unavailable · License not fully assessed Upstream does not publish pretrained DP3 policy checkpoints; only VRL3 RL expert checkpoints for generating Adroit simulation demos are linked. [1] |
| Training datasets | available · License not fully assessed Real-world Franka+Allegro zarr dataset and Adroit/DexArt demo generation assets are linked via Google Drive/OneDrive without a separately declared dataset license. [1] |
| Documentation | available · MIT [1] [3] [5] |
Licensing and weight availability boundaries
The codebase and documentation are released under the MIT License (LICENSE at 47385d9d6f5b), while bundled third_party/ simulation packages retain their respective upstream licenses. Crucially, upstream does not publish pretrained DP3 diffusion policy checkpoints—only VRL3 reinforcement learning expert checkpoints used to generate Adroit demonstrations—and the reviewed materials do not establish a separate dataset license for the linked Google Drive real-robot datasets. [1] [3]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 YanjieZe/3D-Diffusion-Policy GitHub repository and README.md
Documents DP3 and Simple DP3 (~10 GB VRAM / ~3h training on A40; 25 FPS inference for Simple DP3), 57 simulation tasks, real-robot Franka+Allegro+L515 setup, and VRL3 expert / Google Drive data links without pretrained DP3 policy checkpoints.Revision, date & where cited
Yanjie Ze (GitHub) · github.com · revision
47385d9· README.md at commit 47385d9d6f5b (2025-10-17) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 7 more - S002 DP3 Hydra configuration (dp3.yaml)
Specifies PointNet encoder (in_channels: 3, out_channels: 64, use_layernorm: true, use_pc_color: false), 1D U-Net (down_dims: [512, 1024, 2048]), DDIMScheduler (100 train steps, 10 inference steps), horizon: 16, n_obs_steps: 2, n_action_steps: 8.Revision, date & where cited
Yanjie Ze (GitHub) · github.com · revision
47385d9· dp3.yaml at commit 47385d9d6f5b · Accessed Nov 21, 2026
Used for: Model card summary, Compute & fine-tuning, Compact 3D encoder and DDIM action head - S003 YanjieZe/3D-Diffusion-Policy LICENSE
MIT License, Copyright (c) 2024 Yanjie Ze.Revision, date & where cited
Yanjie Ze (GitHub) · github.com · revision
47385d9· LICENSE at commit 47385d9d6f5b · Accessed Nov 21, 2026
Used for: Overview, Separate license matrix - S004 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations (RSS 2024)
Reports 24.2% relative improvement over baselines across 72 simulation tasks with 10 demos and 85% average success rate across 4 real-robot tasks with 40 demos each.Revision, date & where cited
arXiv (2403.03954v7) · arxiv.org · arXiv abstract 2403.03954v7 · Accessed Nov 21, 2026
Used for: Overview, Is this for you?, Upstream-reported benchmarks, Setup constraints and upstream-reported benchmarks - S005 YanjieZe/3D-Diffusion-Policy INSTALL.md
Documents Python 3.8, CUDA 11.7+/12.1, RTX 3090/A40/A800/A100 GPUs, MuJoCo 2.1.0, mujoco-py 2.1.2.14, gym 0.21.0, diffusers 0.11.1, and zarr 2.12.0.Revision, date & where cited
Yanjie Ze (GitHub) · github.com · revision
47385d9· INSTALL.md at commit 47385d9d6f5b · Accessed Nov 21, 2026
Used for: Is this for you?, Compute & fine-tuning, Setup & usage, Setup constraints and upstream-reported benchmarks and 1 more
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile drafted from YanjieZe/3D-Diffusion-Policy@47385d9d6f5b.