Open Source RoboticsRobots, software & learning

3D Diffusion Policy (DP3)

Robotics model

MIT-licensed 3D visuomotor diffusion policy architecture combining a compact PointNet-style sparse point-cloud encoder with a 1D conditional U-Net DDIM action head for data-efficient dexterous and arm manipulation; upstream releases training/evaluation code, RL demonstration experts, and real-robot datasets, but not pretrained DP3 policy weights. [1] [2] [3] [4]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Robot learning researchers; Dexterous manipulation practitioners; Imitation learning engineers using depth cameras [1] [5]
Parameters
Task-dependent (not pinned as a single scalar in README or dp3.yaml [1] [2]
Embodiments
Franka Emika Panda + Allegro Hand (real robot reference), Adroit Shadow Hand (simulation), DexArt articulated-object… [1]
Source rights
Some licenses are unresolved. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware
Ecosystem
MuJoCo
License
MIT · license not fully assessed for datasets · Details by part

Requirements · Research details and dates

On this page

Overview

3D Diffusion Policy (DP3) is an MIT-licensed visuomotor imitation learning architecture from RSS 2024 that pairs a compact 3D point-cloud encoder with a 1D conditional U-Net diffusion policy. Rather than shipping a generalist pretrained checkpoint, the upstream repository (YanjieZe/3D-Diffusion-Policy) provides the reference training and evaluation codebase for task-specific behavior cloning across dexterous hand and arm manipulation tasks, together with simulation benchmark wrappers and real-robot demonstration data. [1] [3] [4]

Model card summary

Architecture

Compact PointNet-style sparse 3D point-cloud encoder (LayerNorm, encoder_output_dim: 64, use_pc_color: false by default) combined with robot proprioception as FiLM global conditioning into a 1D conditional U-Net DDIM diffusion action head (down_dims: [512, 1024, 2048] in dp3.yaml; 100 training timesteps, 10 DDIM inference steps). Also ships a faster Simple DP3 variant (simple_dp3.yaml).

[1] [2]

Parameter count

Task-dependent (not pinned as a single scalar in README or dp3.yaml; uses compact PointNet encoder with 64-D output + 1D U-Net with down_dims [512, 1024, 2048] in dp3.yaml)

Exact parameter count depends on task state/action dimensions and whether dp3.yaml or simple_dp3.yaml is used.

[1] [2] · Editorial assessment

Input modalities
point-cloudproprioception

Sparse 3D point clouds (shape (T, Np, 3) xyz by default with use_pc_color: false, or (T, Np, 6) xyzrgb in real-robot format) plus agent_pos proprioception.

[1] [2]

Output modalities
ee-pose-deltajoint-positions

Real-robot README documents 22-D action space using relative end-effector position control (6-D) for the Franka arm and relative joint-angle position control (16-D) for the Allegro hand; README also notes global position action space can improve results.

[1]

Action space

Continuous DDIM-denoised action chunks (default horizon: 16, n_obs_steps: 2, n_action_steps: 8, 10 DDIM inference steps in dp3.yaml); task-specific dimensions (e.g., 22-D for Franka 6-D arm + Allegro 16-D hand).

[1] [2]

Control frequency

Up to 25 Hz inference reported for Simple DP3 (25 FPS per README)

Full dp3.yaml inference FPS is not given as a single scalar in the README.

[1]

Trained / supported embodiments
  • Franka Emika Panda + Allegro Hand (real robot reference)
  • Adroit Shadow Hand (simulation)
  • DexArt articulated-object manipulation (simulation)
  • MetaWorld Sawyer arm (simulation)

[1]

Training datasets
  • Adroit demonstrations generated from provided VRL3 RL expert checkpoints
  • DexArt demonstrations generated from provided assets
  • MetaWorld demonstrations (50 tasks)
  • Real-world Franka + Allegro zarr demonstration dataset (Google Drive)

[1]

Interfaces
  • Hydra YAML CLI scripts (train_policy.sh, eval_policy.sh)
  • Zarr demonstration datasets
  • ROS Noetic / Frankx / Allegro controller (real robot reference)

[1] [2]

Checkpoints

CheckpointFormatSizeAccessPinned revision
DP3 policy checkpoints (not released upstream; train per task from demonstrations)pytorchNot statedweights-not-releasedNot pinned

Upstream releases VRL3 RL expert checkpoints solely for generating Adroit simulation demonstrations, not pretrained DP3 diffusion policy checkpoints.

[1]

Is this for you?

Use cases
  • Training task-specific 3D visuomotor policies from 10 to 40 demonstrations
  • Benchmarking 3D imitation learning on Adroit, DexArt, and MetaWorld
  • Real-world dexterous manipulation with cropped point clouds from depth cameras

[1] [4]

Known limitations
  • No pretrained DP3 policy checkpoints are published upstream; users must generate demonstrations or supply custom zarr datasets and train per task.
  • Sensitive to depth sensor quality: README warns that Intel RealSense D435 point clouds can cause failure compared to RealSense L515.
  • Requires manual point-cloud bounding-box cropping and FPS downsampling to remove table/background points in real-world deployments.
  • Simulation installation pins legacy Python 3.8, gym 0.21.0, MuJoCo 2.1.0, and mujoco-py 2.1.2.14.

[1] [5]

Safety and limitations
  • No pretrained policy weights are distributed; every target robot/task requires demonstration collection and per-task training.
  • Depth sensor noise directly degrades policy behavior; README warns RealSense D435 point clouds can cause failures.
  • Real-robot deployment code is not self-contained in this repo; README directs users to Improved-3D-Diffusion-Policy (iDP3) for deployment reference.

[1]

Compute & fine-tuning

Inference compute
  • Documented hardware: NVIDIA RTX 3090 / A40 / A800 / A100 (per INSTALL.md and README)
  • Precision: fp32
  • Reported latency: 40 ms

40 ms corresponds to the README's 25 FPS inference figure for simple_dp3.yaml; training uses ~10 GB VRAM on an A40, while exact minimum inference VRAM is not separately stated.

[1] [5] · Editorial assessment

Fine-tuning
  • Supported: Yes
  • Methods: From-scratch per-task behavior cloning on zarr point-cloud datasets (dp3.yaml / simple_dp3.yaml)
  • Minimum VRAM: 10 GB
  • Recipe: Upstream fine-tuning documentation

README reports ~10 GB GPU memory and ~3 hours on an A40 for dp3.yaml (1-2 hours for simple_dp3.yaml).

[1]

Host computer

Tested on NVIDIA RTX 3090 / A40 / A800 / A100 with CUDA 11.7+; README reports ~10 GB GPU memory and ~3 hours training on an A40 for dp3.yaml (1-2 hours for simple_dp3.yaml).

[1] [5]

Operating system
  • Linux (Ubuntu 20.04 tested for real robot)

[1] [5]

Dependencies
  • Python 3.8
  • PyTorch
  • diffusers==0.11.1
  • hydra-core==1.2.0
  • zarr==2.12.0
  • MuJoCo 2.1.0 / mujoco-py-2.1.2.14
  • pytorch3d_simplified (bundled)

[5]

Skills
  • Python
  • PyTorch
  • Point-cloud preprocessing
  • Hydra / WandB

Inferred from INSTALL.md, Hydra YAML configuration, and point-cloud conversion scripts.

[1] [2] [5] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [5] · Model card / documentation [1]

  • Create a Python 3.8 conda environment and install PyTorch matching host CUDA
  • Install 3D-Diffusion-Policy in editable mode (pip install -e .)
  • Install MuJoCo 2.1.0, bundled third_party/mujoco-py-2.1.2.14, gym-0.21.0, DexArt, MetaWorld, and simplified PyTorch3D
  • Download VRL3 Adroit RL experts and DexArt assets or real-robot zarr data, generate demonstrations, and run scripts/train_policy.sh

[1] [5]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
72 simulation tasks (Adroit, DexArt, MetaWorld, etc. in arXiv:2403.03954v7)24.2% relative improvement over baselines with 10 demonstrationsUpstream-reported paper result; repository bundles 57 tasks (3 Adroit + 4 DexArt + 50 MetaWorld) and notes regenerated demonstrations can vary from paper numbers. Not independently verified by OSR.
4 real-robot manipulation tasks (Franka + Allegro Hand + RealSense L515, arXiv:2403.03954v7)85% average success rate with 40 demonstrations per taskUpstream-reported real-robot evaluation using cropped L515 point clouds; not independently verified by OSR.

[1] [4]

Project notes

Compact 3D encoder and DDIM action head

At the pinned commit (47385d9d6f5b), dp3.yaml configures a lightweight PointNet-style encoder (in_channels: 3, out_channels: 64, use_layernorm: true, use_pc_color: false) that compresses sparse 3D point clouds and fuses them with robot proprioception (agent_pos) as FiLM global conditioning into a 1D U-Net (down_dims: [512, 1024, 2048]). Actions are generated with a DDIMScheduler using 100 training timesteps and 10 inference denoising steps (horizon: 16, n_obs_steps: 2, n_action_steps: 8). The repository also provides simple_dp3.yaml, a lighter variant recommended by the authors for faster training (1–2 hours) and inference (up to 25 FPS). Exact parameter counts are not stated as a single scalar in the README or config and depend on task state/action dimensionality and config variant. [1] [2]

Setup constraints and upstream-reported benchmarks

The repository bundles 57 simulation tasks across Adroit (3), DexArt (4), and MetaWorld (50), plus a real-world Franka Emika Panda + Allegro Hand + Intel RealSense L515 demonstration dataset (zarr). Training dp3.yaml requires roughly 10 GB of GPU memory and ~3 hours on an NVIDIA A40, while installation pins legacy simulation dependencies including Python 3.8, gym==0.21.0, MuJoCo 2.1.0, and mujoco-py==2.1.2.14. Upstream warns that lower-quality depth cameras such as the RealSense D435 can cause DP3 failures and recommends cropping table/background points before FPS downsampling. Note also that real-world robot deployment code is not self-contained in this repository; upstream directs readers to Improved-3D-Diffusion-Policy (iDP3) as a deployment reference, so this codebase and dataset should not be treated as a complete hardware deployment package. In arXiv:2403.03954v7, the authors report a 24.2% relative improvement over baselines across 72 simulation tasks using 10 demonstrations and an 85% average success rate across 4 real-robot tasks using 40 demonstrations each. All reported metrics are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [4] [5]

What we checked

None. Prepared from public upstream sources without hands-on training, inference, or robot execution.

Updates

Upstream last updated
Oct 17, 2025 [1]
Latest release
Not established
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
YanjieZe/3D-Diffusion-Policy@47385d9d6f5bde3f2ebdf2400ecb8261cc9e6b97 · software repository

Separate license matrix

MIT-licensed 3D visuomotor diffusion policy codebase with bundled simulation benchmarks and real-robot zarr datasets, included as partial because upstream does not publish pretrained DP3 policy checkpoints and linked Drive datasets lack an explicit separate license.

LayerAvailability / license
Codeavailable · MIT
Root LICENSE is MIT (Copyright 2024 Yanjie Ze); third_party/ bundles external simulation dependencies with their own upstream licenses. [1] [3]
Model weightsunavailable · License not fully assessed
Upstream does not publish pretrained DP3 policy checkpoints; only VRL3 RL expert checkpoints for generating Adroit simulation demos are linked. [1]
Training datasetsavailable · License not fully assessed
Real-world Franka+Allegro zarr dataset and Adroit/DexArt demo generation assets are linked via Google Drive/OneDrive without a separately declared dataset license. [1]
Documentationavailable · MIT
[1] [3] [5]

Licensing and weight availability boundaries

The codebase and documentation are released under the MIT License (LICENSE at 47385d9d6f5b), while bundled third_party/ simulation packages retain their respective upstream licenses. Crucially, upstream does not publish pretrained DP3 diffusion policy checkpoints—only VRL3 reinforcement learning expert checkpoints used to generate Adroit demonstrations—and the reviewed materials do not establish a separate dataset license for the linked Google Drive real-robot datasets. [1] [3]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile drafted from YanjieZe/3D-Diffusion-Policy@47385d9d6f5b.