Open Source RoboticsRobots, software & learning

3D Diffuser Actor

Robotics model

MIT-licensed 3D relative-attention diffusion policy from CMU that lifts multi-view or single-view RGB-D observations into 3D scene tokens to denoise 3D end-effector translations and rotations on RLBench and CALVIN, with six open MIT checkpoints on Hugging Face. [1] [2] [3] [4]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
3D robot manipulation researchers; RLBench and CALVIN benchmark practitioners; Diffusion policy developers [1] [3]
Parameters
Not explicitly stated as a scalar in README or HF model card (DiffuserActor .pth checkpoints range from 0.17 GB /… [3]
Embodiments
Franka Emika Panda (RLBench and CALVIN simulation, plus real-robot demos in paper) [1] [3] [4]
Source rights
Some licenses are unresolved. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware
License
MIT · license not fully assessed for datasets · Details by part

Requirements · Research details and dates

On this page

Overview

3D Diffuser Actor is a language-conditioned 3D manipulation policy from Carnegie Mellon University that combines 3D scene feature representations with conditional action diffusion. Both the standalone Python codebase (nickgkan/3d_diffuser_actor) and six pretrained checkpoints on Hugging Face (katefgroup/3d_diffuser_actor)—covering 3D Diffuser Actor and an Act3D re-implementation—are released under the MIT License. [1] [2] [3]

Model card summary

Architecture

3D relative-attention diffusion transformer policy that lifts single- or multi-view RGB-D observations into 3D scene tokens, conditions on 512-D CLIP text instruction encodings and end-effector proprioception history, and iteratively denoises 3D end-effector translations and continuous 6D rotations (120-D embedding / 100 diffusion steps on RLBench; 192-D embedding / 25 diffusion steps on CALVIN). Also includes an Act3D keypose re-implementation.

[1] [3] [5]

Base model

CLIP Text Encoder (used for 512-D language instruction encoding)

Language instructions are pre-encoded with CLIP into 512-D token embeddings.

[1] [5]

Parameter count

Not explicitly stated as a scalar in README or HF model card (DiffuserActor .pth checkpoints range from 0.17 GB / 0.16 GiB to 0.26 GB / 0.24 GiB)

File byte sizes verified directly via Hugging Face API tree metadata (168,026,070 to 256,564,811 bytes).

[3] · Editorial assessment

Input modalities
rgbpoint-cloudproprioceptionlanguage

Takes rgb_obs (B, N_cam, 3, H, W), pcd_obs (B, N_cam, 3, H, W), 512-D instruction encodings, and curr_gripper history (B, H, 7).

[3] [5]

Output modalities
ee-pose-absgripper-command

Outputs pose trajectories or keyposes of shape (B, T, 8) covering 3D position, rotation (internally denoised as continuous 6D rotation), and gripper openness.

[3] [5]

Action space

8-D end-effector keypose or trajectory sequence (3D translation + 4D quaternion converted from continuous 6D rotation + 1D gripper openness); 100 diffusion steps on RLBench and 25 steps on CALVIN.

[3] [5]

Trained / supported embodiments
  • Franka Emika Panda (RLBench and CALVIN simulation, plus real-robot demos in paper)

[1] [3] [4]

Training datasets
  • RLBench (PerAct 18-task multi-view setup; packaged archive Peract_packaged.zip 44.75 GB / 41.68 GiB on HF)
  • RLBench (GNFactor 10-task single-view setup)
  • CALVIN (ABC->D split)

[1] [3]

Interfaces
  • Standalone Python API (from diffuser_actor import DiffuserActor, Act3D)
  • Bash evaluation and training scripts (RLBench, CALVIN)
  • PyTorch .pth checkpoints on Hugging Face Hub

[1] [3] [5]

Checkpoints

CheckpointFormatSizeAccessPinned revision
diffuser_actor_peract.pth (RLBench PerAct)pytorch0.17 GBopena5a7819af754
diffuser_actor_gnfactor.pth (RLBench GNFactor)pytorch0.17 GBopena5a7819af754
diffuser_actor_calvin.pth (CALVIN w/ history)pytorch0.19 GBopena5a7819af754
diffuser_actor_calvin_nohistory.pth (CALVIN w/o history)pytorch0.26 GBopena5a7819af754
act3d_peract.pth (Act3D RLBench PerAct)pytorch0.17 GBopena5a7819af754
act3d_gnfactor.pth (Act3D RLBench GNFactor)pytorch0.17 GBopena5a7819af754

All six ungated MIT PyTorch checkpoints verified via Hugging Face API.

[1] [3]

Is this for you?

Use cases
  • Multi-view and single-view language-conditioned 3D manipulation on RLBench
  • Zero-shot unseen-scene long-horizon manipulation on CALVIN (ABC->D)
  • Standalone DiffuserActor / Act3D policy integration via pip install -e .

[1] [5]

Known limitations
  • Released checkpoints assume input quaternions in wxyz format whereas training scripts were updated to xyzw; users must match quaternion_format when evaluating released weights vs retraining.
  • Depends on specific legacy packages including dgl==1.1.3+cu116, flash-attn==2.5.9.post1, CoppeliaSim 4.1, and a modified RLBench fork (with a close_jar success-condition fix).
  • Requires calibrated RGB and aligned 3D point clouds plus pre-encoded 512-D CLIP text instructions.

[1] [3] [5]

Safety and limitations
  • Quaternion convention mismatch (wxyz in released checkpoints vs xyzw in updated training scripts) can silently break pose predictions if misconfigured.
  • Trained on simulation benchmarks (RLBench, CALVIN); real-world deployment requires calibrated depth/point clouds and custom robot execution wrappers.

[1] [3]

Compute & fine-tuning

Inference compute
  • Documented hardware: NVIDIA CUDA GPU (CUDA 11.6+ for DGL and FlashAttention)
  • Precision: fp32
  • Edge / consumer capable: No

Requires CUDA GPU for FlashAttention and DGL; exact minimum VRAM and latency_ms are not specified in the README or model card.

[1] · Editorial assessment

Fine-tuning
  • Supported: Yes
  • Methods: From-scratch training and retraining on RLBench or CALVIN via main_keypose.py / main_trajectory.py / main_trajectory_calvin.py
  • Recipe: Upstream fine-tuning documentation

Note: training scripts use xyzw quaternion format whereas released checkpoints expect wxyz.

[1]

Host computer

NVIDIA CUDA GPU required (DGL cu116 and FlashAttention 2.5.9.post1); exact VRAM requirements are not pinned in the README.

Inferred from CUDA-only DGL and FlashAttention dependencies.

[1] · Editorial assessment

Operating system
  • Linux (Ubuntu 20.04 for CoppeliaSim 4.1)

[1]

Dependencies
  • Python 3.8 (conda environment.yaml)
  • PyTorch
  • diffusers[torch]
  • dgl==1.1.3+cu116
  • flash-attn==2.5.9.post1
  • open3d, PyRep, CoppeliaSim 4.1, RLBench (PerAct fork) or CALVIN

[1]

Skills
  • Python
  • PyTorch / diffusers
  • CUDA (DGL, FlashAttention)
  • RLBench / CALVIN simulation

Inferred from documented conda, CUDA wheel, and simulator setup steps.

[1] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1] · Model card / documentation [1]

  • Create the conda environment from environment.yaml and install diffusers[torch], dgl==1.1.3+cu116, and flash-attn==2.5.9.post1
  • Install the standalone model package with pip install -e . (exposing DiffuserActor and Act3D)
  • Install CALVIN or PyRep + CoppeliaSim 4.1 + PerAct RLBench fork for simulation evaluation
  • Download pretrained .pth weights and optional instructions.zip / Peract_packaged.zip from katefgroup/3d_diffuser_actor into train_logs/ and run evaluation scripts

[1] [3]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
RLBench (PerAct 18-task multi-view setup)81.3% average success rate (vs 62.9% RVT baseline in model card)Upstream-reported simulation result from HF model card; README also notes a close_jar RLBench success-condition fix. Not independently verified by OSR.
RLBench (GNFactor 10-task single-view setup)78.4% average success rate (vs 31.7% GNFactor baseline in model card)Upstream-reported simulation result from HF model card; not independently verified by OSR.
CALVIN (ABC->D zero-shot unseen scene generalization)92.2% / 78.7% / 63.9% / 51.2% / 41.2% success across 1 to 5 consecutive tasksUpstream-reported simulation result from HF model card (with-history checkpoint); README notes a newer no-history checkpoint (diffuser_actor_calvin_nohistory.pth) achieves higher performance. Not independently verified by OSR.

[1] [3]

Project notes

3D scene tokens, checkpoint mapping, and quaternion caveat

As documented in docs/OVERVIEW.md (4faf00bdf89a) and the Hugging Face model card (a5a7819af754), the policy takes multi-view or single-view RGB images, aligned 3D point clouds, 512-D CLIP text instruction embeddings, and end-effector proprioception history (curr_gripper), and iteratively denoises 3D end-effector translations and continuous 6D rotations into 8-D pose keyposes or trajectories (using 120-D embeddings and 100 diffusion steps on RLBench, or 192-D embeddings and 25 steps on CALVIN). Exact parameter counts are not stated in the README or model card. For evaluation, download the relevant .pth checkpoint into train_logs/: [1] [3] [5]

Checkpoint fileModelBenchmark / variantSize
diffuser_actor_peract.pth3D Diffuser ActorRLBench (PerAct, 18 tasks)~0.17 GB (0.16 GiB)
diffuser_actor_gnfactor.pth3D Diffuser ActorRLBench (GNFactor, 10 tasks)~0.17 GB (0.16 GiB)
diffuser_actor_calvin.pth3D Diffuser ActorCALVIN (with history)~0.19 GB (0.18 GiB)
diffuser_actor_calvin_nohistory.pth3D Diffuser ActorCALVIN (without history, Aug 2024)~0.26 GB (0.24 GiB)
act3d_peract.pthAct3DRLBench (PerAct)~0.17 GB (0.16 GiB)
act3d_gnfactor.pthAct3DRLBench (GNFactor)~0.17 GB (0.16 GiB)

Upstream highlights an important compatibility caveat: the released 3D Diffuser Actor checkpoints assume wxyz input quaternions, whereas the updated training scripts use xyzw, so evaluators must pass the matching quaternion_format flag. [1] [3] [5]

Dependencies and upstream-reported benchmarks

While DiffuserActor and Act3D can be installed as a standalone Python package (pip install -e .), full reproduction depends on a CUDA 11.6 wheel for dgl==1.1.3+cu116, a separate pinned installation of flash-attn==2.5.9.post1 --no-build-isolation, and either CALVIN or CoppeliaSim 4.1 with the PerAct RLBench fork. Minimum VRAM and inference latency are not established by the inspected documentation, and checkpoint file sizes do not establish runtime memory requirements. For RLBench evaluation, the README also instructs users to patch the close_jar task success condition, which is incorrect in the upstream RLBench fork. On the Hugging Face model card, upstream reports 81.3% average success on the 18-task RLBench PerAct multi-view benchmark (vs 62.9% for RVT), 78.4% average success on the 10-task RLBench GNFactor single-view benchmark (vs 31.7% for GNFactor—note that the model card's GNFactor table contains an apparent task-row typo of 894.7 for sweep to dustpan, so 78.4% is retained strictly as the upstream-reported average), and—for the with-history CALVIN checkpoint—92.2% / 78.7% / 63.9% / 51.2% / 41.2% success across 1–5 consecutive tasks on CALVIN ABC→D zero-shot unseen-scene evaluation (upstream notes the newer diffuser_actor_calvin_nohistory.pth performs better, without publishing a replacement table in the model card). All scores are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [3]

MIT code and weights, with unresolved archive dataset terms

3D Diffuser Actor carries permissive MIT licensing for both its GitHub codebase (LICENSE at 4faf00bdf89a) and its six Hugging Face model checkpoints (a5a7819af754). The Hugging Face repository also hosts downloadable Peract_packaged.zip (~44.75 GB / 41.68 GiB) and pre-encoded instructions.zip (~0.88 GB / 0.82 GiB), whose underlying RLBench/PerAct and CALVIN dataset license terms are not separately established by the inspected files and remain unresolved here. Repository activity has been dormant since August 2024, so users targeting newer PyTorch or CUDA stacks should expect dependency adaptation. [1] [2] [3]

What we checked

None. Prepared from public upstream sources without hands-on training, inference, or robot execution.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Aug 17, 2024 [1]
Latest release
Aug 16, 2024 [1] [3]
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
nickgkan/3d_diffuser_actor@4faf00bdf89a3b3d07e266a59a28f9fbd02c7844; HF katefgroup/3d_diffuser_actor@a5a7819af75491d0e7d9e606f635cc3f95cb3a1c · combined hardware and software repository

Separate license matrix

Full MIT-licensed codebase and six ungated MIT pretrained checkpoints (3D Diffuser Actor and Act3D) on Hugging Face; classified as partial because underlying dataset license terms for the hosted RLBench/CALVIN archives are not separately established in the inspected files.

LayerAvailability / license
Codeavailable · MIT
[1] [2]
Model weightsavailable · MIT
Model card YAML declares license: mit and body confirms 'The code and model are released under MIT License'. [3]
Training datasetsavailable · License not fully assessed
Hosts Peract_packaged.zip (44.75 GB / 41.68 GiB) and instructions.zip (0.88 GB / 0.82 GiB) in the HF model repo; underlying RLBench/PerAct and CALVIN dataset license terms are not separately established by the inspected files. [1] [3]
Documentationavailable · MIT
[2] [5]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile drafted from nickgkan/3d_diffuser_actor@4faf00bdf89a and katefgroup/3d_diffuser_actor@a5a7819af754.