DexVLA
Robotics modelVision-Language-Action architecture pairing an off-the-shelf Qwen2-VL-2B-Instruct backbone with a plug-in ScaleDP diffusion transformer action expert (~1B ScaleDP-H or ~400M/410M ScaleDP-L), releasing Stage-1 pretrained ScaleDP weights (CC-BY-NC-4.0) and three-stage training code under Attribution-NonCommercial terms. [1] [2] [3] [4]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Robotics and VLA researchers; Bimanual manipulation practitioners (ALOHA / AgileX-style HDF5 pipelines) [1]
- Parameters
- ~3B combined (~2B Qwen2-VL-2B-Instruct VLM + ~1B ScaleDP-H policy head [1]
- Embodiments
- AgileX / ALOHA-style bimanual arms (evaluate/smart_eval_agilex.py, aloha_scripts/), Custom multi-camera HDF5 /… [1]
- Source rights
- Named licenses are documented by part. Includes noncommercial terms. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- License
- Creative Commons Attribution-NonCommercial summary terms (version not stated in root LICENSE; non-OSI) (noncommercial), CC-BY-NC-4.0 (ScaleDP-H and ScaleDP-L); separate upstream terms apply to Qwen/Qwen2-VL-2B-Instruct (noncommercial), Not specified in README for example dataset; full Stage-1/Stage-2 training corpus is not bundled, Creative Commons Attribution-NonCommercial summary terms (version not stated in root LICENSE) (noncommercial) · Details by part
Overview
DexVLA is a Vision-Language-Action (VLA) architecture that combines an off-the-shelf Qwen/Qwen2-VL-2B-Instruct vision-language backbone with a plug-in ScaleDP diffusion transformer action expert (~1B ScaleDP-H or ~400M/410M ScaleDP-L), yielding roughly 3B combined parameters in the full configuration. Rather than releasing merged end-to-end robot weights, the authors publish the Stage-1 pretrained ScaleDP policy-head checkpoints (lesjie/scale_dp_h and lesjie/scale_dp_l) alongside the three-stage PyTorch training and evaluation codebase (juruobenruo/DexVLA at commit fc21a822f4c7). [1] [3] [4] [5]
Model card summary
- Architecture
Qwen2-VL-2B-Instruct vision-language backbone (28 layers, hidden_size 1536, ViT depth 32) paired with a plug-in ScaleDP diffusion transformer action expert (1B ScaleDP-H or 410M ScaleDP-L) using DDIM denoising, plus optional substep reasoning tokens.
- Base model
Qwen/Qwen2-VL-2B-Instruct (VLM backbone) + ScaleDP-H / ScaleDP-L (Stage-1 diffusion policy head)
- Parameter count
~3B combined (~2B Qwen2-VL-2B-Instruct VLM + ~1B ScaleDP-H policy head; or ~2.4B with ~400M/410M ScaleDP-L)
README reports ~2B Qwen2-VL, ~1B ScaleDP-H, and ~400M (table) / 410M (Smaller ScaleDP section) ScaleDP-L.
- Input modalities
- rgbproprioceptionlanguage
Multi-view RGB cameras (e.g., left, right, wrist at 480x640x3 in example HDF5), robot joint_positions/qpos/qvel proprioception, raw language instructions, and optional substep reasoning.
- Output modalities
- joint-positionsgripper-command
Continuous diffused action chunks (e.g., 10-D action vectors in example HDF5 schema; AgileX/ALOHA bimanual arm/gripper control).
- Action space
Continuous diffusion-denoised action chunks generated by the ScaleDP transformer policy head (DDIMScheduler); example HDF5 schema uses 10-D action vectors with 7-D joint_positions/qpos.
- Trained / supported embodiments
- AgileX / ALOHA-style bimanual arms (evaluate/smart_eval_agilex.py, aloha_scripts/)
- Custom multi-camera HDF5 / RLDS-converted embodiments
- Training datasets
- Multi-embodiment Stage-1 ScaleDP pretraining data (upstream details in arXiv:2502.05855)
- Example dataset: lesjie/dexvla_example_data (Hugging Face)
- User-supplied ACT-format HDF5 or RLDS-converted demonstrations for Stage 2 / Stage 3
- Interfaces
- PyTorch / Hugging Face Transformers custom architecture (Qwen2VLForConditionalGenerationForVLA)
- Shell training scripts (DeepSpeed / Python)
- ACT-compatible HDF5 dataset format
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| lesjie/scale_dp_h (Stage-1 ScaleDP-H ~1B policy head) | pytorch | 3.8 GB | open | dbc18619a5c5 |
| lesjie/scale_dp_l (Stage-1 ScaleDP-L ~400M policy head) | pytorch | 1.83 GB | open | 52c2e81af0a5 |
Both checkpoints are ungated Stage-1 diffusion policy-head backbones (open_scale_dp_h_backbone.ckpt at 3.799 GB and open_scale_dp_l_backbone.ckpt at 1.827 GB) under CC-BY-NC-4.0; VLM backbone Qwen/Qwen2-VL-2B-Instruct is downloaded separately.
Is this for you?
- Use cases
- Three-stage VLA training with a pretrained ScaleDP diffusion policy head
- LoRA or full fine-tuning of Qwen2-VL-2B with a diffusion action expert on custom HDF5 robot demonstrations
- Diffusion-VLA (DiVLA) UNet-head baseline training via scripts/train_divla.sh
- Known limitations
- Only Stage-1 pretrained ScaleDP policy-head checkpoints (ScaleDP-H and ScaleDP-L) are published; no merged Stage-2 or Stage-3 full DexVLA weights are released on Hugging Face.
- Repository code and Stage-1 ScaleDP checkpoints are restricted to non-commercial use under CC-BY-NC-4.0.
- Requires replacing Qwen2-VL-2B-Instruct's official config.json with docs/config.json and manually patching diffusers DDIMScheduler to float32 to avoid NaN actions during inference.
- Safety and limitations
- CC-BY-NC-4.0 license on both the DexVLA repository and Stage-1 ScaleDP checkpoints prohibits commercial use.
- No ready-to-deploy full Stage-2/Stage-3 DexVLA checkpoint is published; users must assemble Qwen2-VL-2B-Instruct with Stage-1 ScaleDP weights and train on target demonstrations.
- Known precision overflow in diffusers DDIMScheduler can emit NaN robot actions unless self.betas is cast to float32.
Compute & fine-tuning
- Inference compute
- Documented hardware: NVIDIA A6000 (46 GB VRAM, documented for training benchmarks)
- Precision:
bf16 - Edge / consumer capable: No
docs/config.json specifies torch_dtype bfloat16; README documents 20-38 GB VRAM for training on an A6000, while exact minimum inference VRAM and latency_ms are not reported.
[1] · Editorial assessment
- Fine-tuning
- Supported: Yes
- Methods: Stage-2 multi-task fine-tuning (scripts/train_dexvla_stage2.sh); Stage-3 target dexterous post-training (scripts/train_dexvla_stage3.sh); LoRA on VLM backbone with full policy-head training; DeepSpeed ZeRO-2 with optional CPU optimizer offload
- Minimum VRAM: 20 GB
- Recipe: Upstream fine-tuning documentation
README reports 20-29 GB CUDA memory with DeepSpeed ZeRO-2 CPU offload, 24 GB with LoRA VLM, 33 GB with ScaleDP-L, and 38 GB full on a single A6000 (batch_size 2).
- Host computer
README documents single NVIDIA A6000 (46 GB VRAM) training at batch size 2 requiring 38 GB VRAM (full ScaleDP-H), 33 GB with smaller ScaleDP-L, 24 GB with LoRA VLM, or 20-29 GB with DeepSpeed ZeRO-2 CPU offload.
- Operating system
- Linux
CUDA, flash-attn, GCC > 9 DeepSpeed offload, and shell scripts target Linux.
[1] · Editorial assessment
- Dependencies
- Python 3.10
- PyTorch
- transformers (4.41.2 in docs/config.json)
- diffusers
- flash-attn
- DeepSpeed (optional offload)
- Qwen/Qwen2-VL-2B-Instruct
- Skills
- Python
- PyTorch
- DeepSpeed
- HDF5 / RLDS data preprocessing
Inferred from documented conda, PyTorch, DeepSpeed ZeRO-2, and h5py pipeline.
[1] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [1]
- Clone juruobenruo/DexVLA, create a Python 3.10 conda environment, install requirements.txt, editable-install policy_heads, and install flash-attn.
- Download Qwen/Qwen2-VL-2B-Instruct and replace its config.json with docs/config.json.
- Download Stage-1 ScaleDP-H or ScaleDP-L weights from Hugging Face (lesjie/scale_dp_h or lesjie/scale_dp_l).
- Convert robot demonstrations to ACT-style h5py files, register paths in aloha_scripts/constants.py, and run scripts/train_dexvla_stage2.sh and scripts/train_dexvla_stage3.sh.
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| DexVLA paper evaluations (arXiv:2502.05855; bimanual and dexterous manipulation tasks) | See arXiv:2502.05855 (no numeric task table in repository README) | Upstream-reported paper evaluations; not independently verified or reproduced by OSR. |
Project notes
Architecture, inputs, and three-stage training recipe
In docs/config.json, DexVLA registers a custom Qwen2VLForConditionalGenerationForVLA architecture (hidden_size: 1536, 28 layers, torch_dtype: bfloat16) with policy_head_type: "scale_dp_policy". Data pipelines follow the ACT-compatible HDF5 structure (with an RLDS-to-HDF5 conversion script provided), ingesting multi-view RGB images (for example left, right, and wrist views at 480x640x3), proprioceptive state (joint_positions, qpos, qvel), raw language instructions, and optional substep reasoning strings to generate continuous diffusion-denoised action chunks. Training proceeds from Stage-1 ScaleDP pretraining into Stage-2 multi-task training (scripts/train_dexvla_stage2.sh) and Stage-3 target dexterous post-training (scripts/train_dexvla_stage3.sh); the repository also includes a script for the UNet-head Diffusion-VLA (DiVLA) predecessor. [1]
Compute footprint, caveats, and upstream-reported evaluations
The README benchmarks single-GPU training on an NVIDIA A6000 (46 GB VRAM, batch size 2): full training consumes 38 GB VRAM (1.1s/iter), switching to ScaleDP-L lowers usage to 33 GB, LoRA fine-tuning on the VLM backbone (with full policy-head training) uses 24 GB, and DeepSpeed ZeRO-2 CPU optimizer offload brings usage to 20–29 GB (6.56s/iter). The README does not establish minimum inference-only VRAM or a fixed closed-loop control frequency. Users must replace Qwen2-VL-2B-Instruct's config.json with docs/config.json, copy preprocessor_config.json and chat_template.json into trained checkpoints, and apply a documented float32 cast in diffusers's DDIMScheduler to avoid NaN actions during denoising. Benchmark evaluations are reported in arXiv:2502.05855 (no numeric task table appears in the README); Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [5]
What we checked
None. Prepared from public upstream sources without hands-on training, fine-tuning, or robot execution.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Upstream last updated
- Apr 15, 2025 [1]
- Latest release
- Feb 28, 2025 [4]
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- juruobenruo/DexVLA@fc21a822f4c774e242eb6f1ab4a235788de7aba9; HF lesjie/scale_dp_h@dbc18619a5c575488c1c0f09d0ced11e01853d3c; HF lesjie/scale_dp_l@52c2e81af0a5f45bf286be0fe0fc10bfdac636c1 · combined hardware and software repository
Separate license matrix
Publishes reproducible VLA training/evaluation code and ungated Stage-1 ScaleDP diffusion policy-head checkpoints (ScaleDP-H ~1B and ScaleDP-L ~400M), but both code and weights carry a non-commercial CC-BY-NC-4.0 restriction and full Stage-2/Stage-3 merged weights are not published.
| Layer | Availability / license |
|---|---|
| Code | available · Creative Commons Attribution-NonCommercial summary terms (version not stated in root LICENSE; non-OSI) Root LICENSE contains the Creative Commons Attribution-NonCommercial deed summary (Share, Adapt, Attribution, NonCommercial, No additional restrictions) without a version number; GitHub API returns NOASSERTION. [1] [2] |
| Model weights | available · CC-BY-NC-4.0 (ScaleDP-H and ScaleDP-L); separate upstream terms apply to Qwen/Qwen2-VL-2B-Instruct Stage-1 ScaleDP-H and ScaleDP-L weights are ungated on Hugging Face under cc-by-nc-4.0; full Stage-2/Stage-3 merged DexVLA checkpoints are not released. [1] [3] [4] |
| Training datasets | available · Not specified in README for example dataset; full Stage-1/Stage-2 training corpus is not bundled An example HDF5 dataset is linked on Hugging Face for pipeline testing, while full pretraining data is not released. [1] |
| Documentation | available · Creative Commons Attribution-NonCommercial summary terms (version not stated in root LICENSE) [1] [2] |
Non-commercial licensing and partial weight availability
The juruobenruo/DexVLA repository root LICENSE contains Creative Commons Attribution-NonCommercial summary terms (Share, Adapt, Attribution, NonCommercial, No additional restrictions) without stating a version identifier, while the two ungated Hugging Face Stage-1 checkpoints (lesjie/scale_dp_h at 3.80 GB and lesjie/scale_dp_l at 1.83 GB) explicitly declare cc-by-nc-4.0. The separately downloaded Qwen/Qwen2-VL-2B-Instruct VLM backbone carries its own upstream license terms that must be verified separately. Because no Stage-2 or Stage-3 merged DexVLA weights are published, practitioners need suitable demonstrations for target-task adaptation, while the linked example HDF5 dataset (lesjie/dexvla_example_data) can support pipeline testing subject to separate verification of its usage rights (the README does not specify its license) alongside the non-commercial restrictions on both code and Stage-1 weights. [1] [2] [3] [4]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 juruobenruo/DexVLA GitHub repository README.md and docs/config.json
Documents Qwen2-VL-2B-Instruct backbone + ScaleDP-H (~1B) / ScaleDP-L (~400M/410M) diffusion expert, three-stage training recipe, A6000 20-38 GB VRAM benchmarks, DDIMScheduler NaN workaround, and ACT-style HDF5 schema.Revision, date & where cited
juruobenruo (GitHub) · github.com · revision
fc21a82· README.md and docs/config.json at commit fc21a822f4c7 (2025-04-15) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Compute & fine-tuning and 6 more - S002 juruobenruo/DexVLA root LICENSE file
Contains Creative Commons Attribution-NonCommercial summary terms (Share, Adapt, Attribution, NonCommercial, No additional restrictions).Revision, date & where cited
juruobenruo (GitHub) · github.com · revision
fc21a82· LICENSE at commit fc21a822f4c7 · Accessed Nov 21, 2026
Used for: Is this for you?, Separate license matrix - S003 Hugging Face API metadata: lesjie/scale_dp_h
Ungated Stage-1 ScaleDP-H checkpoint (open_scale_dp_h_backbone.ckpt, 3.799 GB), license: cc-by-nc-4.0.Revision, date & where cited
Hugging Face (lesjie) · huggingface.co · HF API at revision dbc18619a5c5 (2025-02-24) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 1 more - S004 Hugging Face API metadata: lesjie/scale_dp_l
Ungated Stage-1 ScaleDP-L checkpoint (open_scale_dp_l_backbone.ckpt, 1.827 GB), license: cc-by-nc-4.0.Revision, date & where cited
Hugging Face (lesjie) · huggingface.co · HF API at revision 52c2e81af0a5 (2025-02-28) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 2 more - S005 arXiv:2502.05855 — DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Control
Introduces DexVLA with a plug-in ScaleDP diffusion expert paired with a vision-language model backbone.Revision, date & where cited
arXiv · arxiv.org · arXiv:2502.05855v3 (2025-02-09) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Upstream-reported benchmarks, Compute footprint, caveats, and upstream-reported evaluations
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from juruobenruo/DexVLA@fc21a822f4c7 and HF Stage-1 ScaleDP checkpoints.