3D Diffuser Actor
Robotics modelMIT-licensed 3D relative-attention diffusion policy from CMU that lifts multi-view or single-view RGB-D observations into 3D scene tokens to denoise 3D end-effector translations and rotations on RLBench and CALVIN, with six open MIT checkpoints on Hugging Face. [1] [2] [3] [4]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- 3D robot manipulation researchers; RLBench and CALVIN benchmark practitioners; Diffusion policy developers [1] [3]
- Parameters
- Not explicitly stated as a scalar in README or HF model card (DiffuserActor .pth checkpoints range from 0.17 GB /… [3]
- Embodiments
- Franka Emika Panda (RLBench and CALVIN simulation, plus real-robot demos in paper) [1] [3] [4]
- Source rights
- Some licenses are unresolved. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- License
- MIT · license not fully assessed for datasets · Details by part
Overview
3D Diffuser Actor is a language-conditioned 3D manipulation policy from Carnegie Mellon University that combines 3D scene feature representations with conditional action diffusion. Both the standalone Python codebase (nickgkan/3d_diffuser_actor) and six pretrained checkpoints on Hugging Face (katefgroup/3d_diffuser_actor)—covering 3D Diffuser Actor and an Act3D re-implementation—are released under the MIT License. [1] [2] [3]
Model card summary
- Architecture
3D relative-attention diffusion transformer policy that lifts single- or multi-view RGB-D observations into 3D scene tokens, conditions on 512-D CLIP text instruction encodings and end-effector proprioception history, and iteratively denoises 3D end-effector translations and continuous 6D rotations (120-D embedding / 100 diffusion steps on RLBench; 192-D embedding / 25 diffusion steps on CALVIN). Also includes an Act3D keypose re-implementation.
- Base model
CLIP Text Encoder (used for 512-D language instruction encoding)
Language instructions are pre-encoded with CLIP into 512-D token embeddings.
- Parameter count
Not explicitly stated as a scalar in README or HF model card (DiffuserActor .pth checkpoints range from 0.17 GB / 0.16 GiB to 0.26 GB / 0.24 GiB)
File byte sizes verified directly via Hugging Face API tree metadata (168,026,070 to 256,564,811 bytes).
[3] · Editorial assessment
- Input modalities
- rgbpoint-cloudproprioceptionlanguage
Takes rgb_obs (B, N_cam, 3, H, W), pcd_obs (B, N_cam, 3, H, W), 512-D instruction encodings, and curr_gripper history (B, H, 7).
- Output modalities
- ee-pose-absgripper-command
Outputs pose trajectories or keyposes of shape (B, T, 8) covering 3D position, rotation (internally denoised as continuous 6D rotation), and gripper openness.
- Action space
8-D end-effector keypose or trajectory sequence (3D translation + 4D quaternion converted from continuous 6D rotation + 1D gripper openness); 100 diffusion steps on RLBench and 25 steps on CALVIN.
- Trained / supported embodiments
- Franka Emika Panda (RLBench and CALVIN simulation, plus real-robot demos in paper)
- Training datasets
- RLBench (PerAct 18-task multi-view setup; packaged archive Peract_packaged.zip 44.75 GB / 41.68 GiB on HF)
- RLBench (GNFactor 10-task single-view setup)
- CALVIN (ABC->D split)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| diffuser_actor_peract.pth (RLBench PerAct) | pytorch | 0.17 GB | open | a5a7819af754 |
| diffuser_actor_gnfactor.pth (RLBench GNFactor) | pytorch | 0.17 GB | open | a5a7819af754 |
| diffuser_actor_calvin.pth (CALVIN w/ history) | pytorch | 0.19 GB | open | a5a7819af754 |
| diffuser_actor_calvin_nohistory.pth (CALVIN w/o history) | pytorch | 0.26 GB | open | a5a7819af754 |
| act3d_peract.pth (Act3D RLBench PerAct) | pytorch | 0.17 GB | open | a5a7819af754 |
| act3d_gnfactor.pth (Act3D RLBench GNFactor) | pytorch | 0.17 GB | open | a5a7819af754 |
All six ungated MIT PyTorch checkpoints verified via Hugging Face API.
Is this for you?
- Use cases
- Multi-view and single-view language-conditioned 3D manipulation on RLBench
- Zero-shot unseen-scene long-horizon manipulation on CALVIN (ABC->D)
- Standalone DiffuserActor / Act3D policy integration via pip install -e .
- Known limitations
- Released checkpoints assume input quaternions in wxyz format whereas training scripts were updated to xyzw; users must match quaternion_format when evaluating released weights vs retraining.
- Depends on specific legacy packages including dgl==1.1.3+cu116, flash-attn==2.5.9.post1, CoppeliaSim 4.1, and a modified RLBench fork (with a close_jar success-condition fix).
- Requires calibrated RGB and aligned 3D point clouds plus pre-encoded 512-D CLIP text instructions.
- Safety and limitations
- Quaternion convention mismatch (wxyz in released checkpoints vs xyzw in updated training scripts) can silently break pose predictions if misconfigured.
- Trained on simulation benchmarks (RLBench, CALVIN); real-world deployment requires calibrated depth/point clouds and custom robot execution wrappers.
Compute & fine-tuning
- Inference compute
- Documented hardware: NVIDIA CUDA GPU (CUDA 11.6+ for DGL and FlashAttention)
- Precision:
fp32 - Edge / consumer capable: No
Requires CUDA GPU for FlashAttention and DGL; exact minimum VRAM and latency_ms are not specified in the README or model card.
[1] · Editorial assessment
- Fine-tuning
- Supported: Yes
- Methods: From-scratch training and retraining on RLBench or CALVIN via main_keypose.py / main_trajectory.py / main_trajectory_calvin.py
- Recipe: Upstream fine-tuning documentation
Note: training scripts use xyzw quaternion format whereas released checkpoints expect wxyz.
- Host computer
NVIDIA CUDA GPU required (DGL cu116 and FlashAttention 2.5.9.post1); exact VRAM requirements are not pinned in the README.
Inferred from CUDA-only DGL and FlashAttention dependencies.
[1] · Editorial assessment
- Operating system
- Linux (Ubuntu 20.04 for CoppeliaSim 4.1)
- Dependencies
- Python 3.8 (conda environment.yaml)
- PyTorch
- diffusers[torch]
- dgl==1.1.3+cu116
- flash-attn==2.5.9.post1
- open3d, PyRep, CoppeliaSim 4.1, RLBench (PerAct fork) or CALVIN
- Skills
- Python
- PyTorch / diffusers
- CUDA (DGL, FlashAttention)
- RLBench / CALVIN simulation
Inferred from documented conda, CUDA wheel, and simulator setup steps.
[1] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [1]
- Create the conda environment from environment.yaml and install diffusers[torch], dgl==1.1.3+cu116, and flash-attn==2.5.9.post1
- Install the standalone model package with pip install -e . (exposing DiffuserActor and Act3D)
- Install CALVIN or PyRep + CoppeliaSim 4.1 + PerAct RLBench fork for simulation evaluation
- Download pretrained .pth weights and optional instructions.zip / Peract_packaged.zip from katefgroup/3d_diffuser_actor into train_logs/ and run evaluation scripts
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| RLBench (PerAct 18-task multi-view setup) | 81.3% average success rate (vs 62.9% RVT baseline in model card) | Upstream-reported simulation result from HF model card; README also notes a close_jar RLBench success-condition fix. Not independently verified by OSR. |
| RLBench (GNFactor 10-task single-view setup) | 78.4% average success rate (vs 31.7% GNFactor baseline in model card) | Upstream-reported simulation result from HF model card; not independently verified by OSR. |
| CALVIN (ABC->D zero-shot unseen scene generalization) | 92.2% / 78.7% / 63.9% / 51.2% / 41.2% success across 1 to 5 consecutive tasks | Upstream-reported simulation result from HF model card (with-history checkpoint); README notes a newer no-history checkpoint (diffuser_actor_calvin_nohistory.pth) achieves higher performance. Not independently verified by OSR. |
Project notes
3D scene tokens, checkpoint mapping, and quaternion caveat
As documented in docs/OVERVIEW.md (4faf00bdf89a) and the Hugging Face model card (a5a7819af754), the policy takes multi-view or single-view RGB images, aligned 3D point clouds, 512-D CLIP text instruction embeddings, and end-effector proprioception history (curr_gripper), and iteratively denoises 3D end-effector translations and continuous 6D rotations into 8-D pose keyposes or trajectories (using 120-D embeddings and 100 diffusion steps on RLBench, or 192-D embeddings and 25 steps on CALVIN). Exact parameter counts are not stated in the README or model card. For evaluation, download the relevant .pth checkpoint into train_logs/: [1] [3] [5]
| Checkpoint file | Model | Benchmark / variant | Size |
|---|---|---|---|
diffuser_actor_peract.pth | 3D Diffuser Actor | RLBench (PerAct, 18 tasks) | ~0.17 GB (0.16 GiB) |
diffuser_actor_gnfactor.pth | 3D Diffuser Actor | RLBench (GNFactor, 10 tasks) | ~0.17 GB (0.16 GiB) |
diffuser_actor_calvin.pth | 3D Diffuser Actor | CALVIN (with history) | ~0.19 GB (0.18 GiB) |
diffuser_actor_calvin_nohistory.pth | 3D Diffuser Actor | CALVIN (without history, Aug 2024) | ~0.26 GB (0.24 GiB) |
act3d_peract.pth | Act3D | RLBench (PerAct) | ~0.17 GB (0.16 GiB) |
act3d_gnfactor.pth | Act3D | RLBench (GNFactor) | ~0.17 GB (0.16 GiB) |
Upstream highlights an important compatibility caveat: the released 3D Diffuser Actor checkpoints assume wxyz input quaternions, whereas the updated training scripts use xyzw, so evaluators must pass the matching quaternion_format flag. [1] [3] [5]
Dependencies and upstream-reported benchmarks
While DiffuserActor and Act3D can be installed as a standalone Python package (pip install -e .), full reproduction depends on a CUDA 11.6 wheel for dgl==1.1.3+cu116, a separate pinned installation of flash-attn==2.5.9.post1 --no-build-isolation, and either CALVIN or CoppeliaSim 4.1 with the PerAct RLBench fork. Minimum VRAM and inference latency are not established by the inspected documentation, and checkpoint file sizes do not establish runtime memory requirements. For RLBench evaluation, the README also instructs users to patch the close_jar task success condition, which is incorrect in the upstream RLBench fork. On the Hugging Face model card, upstream reports 81.3% average success on the 18-task RLBench PerAct multi-view benchmark (vs 62.9% for RVT), 78.4% average success on the 10-task RLBench GNFactor single-view benchmark (vs 31.7% for GNFactor—note that the model card's GNFactor table contains an apparent task-row typo of 894.7 for sweep to dustpan, so 78.4% is retained strictly as the upstream-reported average), and—for the with-history CALVIN checkpoint—92.2% / 78.7% / 63.9% / 51.2% / 41.2% success across 1–5 consecutive tasks on CALVIN ABC→D zero-shot unseen-scene evaluation (upstream notes the newer diffuser_actor_calvin_nohistory.pth performs better, without publishing a replacement table in the model card). All scores are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [3]
MIT code and weights, with unresolved archive dataset terms
3D Diffuser Actor carries permissive MIT licensing for both its GitHub codebase (LICENSE at 4faf00bdf89a) and its six Hugging Face model checkpoints (a5a7819af754). The Hugging Face repository also hosts downloadable Peract_packaged.zip (~44.75 GB / 41.68 GiB) and pre-encoded instructions.zip (~0.88 GB / 0.82 GiB), whose underlying RLBench/PerAct and CALVIN dataset license terms are not separately established by the inspected files and remain unresolved here. Repository activity has been dormant since August 2024, so users targeting newer PyTorch or CUDA stacks should expect dependency adaptation. [1] [2] [3]
What we checked
None. Prepared from public upstream sources without hands-on training, inference, or robot execution.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Upstream last updated
- Aug 17, 2024 [1]
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- nickgkan/3d_diffuser_actor@4faf00bdf89a3b3d07e266a59a28f9fbd02c7844; HF katefgroup/3d_diffuser_actor@a5a7819af75491d0e7d9e606f635cc3f95cb3a1c · combined hardware and software repository
Separate license matrix
Full MIT-licensed codebase and six ungated MIT pretrained checkpoints (3D Diffuser Actor and Act3D) on Hugging Face; classified as partial because underlying dataset license terms for the hosted RLBench/CALVIN archives are not separately established in the inspected files.
| Layer | Availability / license |
|---|---|
| Code | available · MIT [1] [2] |
| Model weights | available · MIT Model card YAML declares license: mit and body confirms 'The code and model are released under MIT License'. [3] |
| Training datasets | available · License not fully assessed Hosts Peract_packaged.zip (44.75 GB / 41.68 GiB) and instructions.zip (0.88 GB / 0.82 GiB) in the HF model repo; underlying RLBench/PerAct and CALVIN dataset license terms are not separately established by the inspected files. [1] [3] |
| Documentation | available · MIT [2] [5] |
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 nickgkan/3d_diffuser_actor GitHub repository and README.md
Documents standalone pip install, DGL/FlashAttention/RLBench/CALVIN dependencies, 6 Hugging Face checkpoints, and wxyz vs xyzw quaternion format caveat.Revision, date & where cited
Nikolaos Gkanatsios / CMU (GitHub) · github.com · revision
4faf00b· README.md at commit 4faf00bdf89a (2024-08-17) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 8 more - S002 nickgkan/3d_diffuser_actor LICENSE
MIT License, Copyright (c) 2024 Tsung-Wei Ke, Nikolaos Gkanatsios and Katerina Fragkiadaki.Revision, date & where cited
Tsung-Wei Ke, Nikolaos Gkanatsios, Katerina Fragkiadaki · github.com · revision
4faf00b· LICENSE at commit 4faf00bdf89a · Accessed Nov 21, 2026
Used for: Overview, MIT code and weights, with unresolved archive dataset terms, Separate license matrix - S003 Hugging Face model card and API metadata: katefgroup/3d_diffuser_actor
Hosts 6 ungated MIT .pth checkpoints (0.17–0.26 GB / 0.16–0.24 GiB), Peract_packaged.zip (44.75 GB / 41.68 GiB), instructions.zip (0.88 GB / 0.82 GiB), and RLBench/CALVIN benchmark tables.Revision, date & where cited
CMU Katerina Group (Hugging Face) · huggingface.co · HF API & README.md at revision a5a7819af754 (2024-08-16) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 7 more - S004 3D Diffuser Actor: Policy Diffusion with 3D Scene Representations
Describes 3D scene featurization, 3D relative attention denoising of translations and rotations, and RLBench/CALVIN/real-robot results.Revision, date & where cited
arXiv (2402.10885v3) · arxiv.org · arXiv abstract 2402.10885v3 · Accessed Nov 21, 2026
Used for: Model card summary - S005 3D Diffuser Actor model overview (docs/OVERVIEW.md)
Details input tensor shapes (RGB, point cloud, 512-D instruction encodings, curr_gripper), continuous 6D rotation representation, and 8-D output pose trajectory format.Revision, date & where cited
Nikolaos Gkanatsios / CMU (GitHub) · github.com · revision
4faf00b· docs/OVERVIEW.md at commit 4faf00bdf89a · Accessed Nov 21, 2026
Used for: Model card summary, Is this for you?, 3D scene tokens, checkpoint mapping, and quaternion caveat, Separate license matrix
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile drafted from nickgkan/3d_diffuser_actor@4faf00bdf89a and katefgroup/3d_diffuser_actor@a5a7819af754.