RVT-2 (Robotic View Transformer 2)
Robotics modelTwo-stage coarse-to-fine multi-view 3D transformer policy from NVIDIA Research (RSS 2024) for language-conditioned, high-precision keyframe manipulation on RLBench and real robots, with source-available non-commercial code and open-download non-commercial checkpoints on Hugging Face. [1] [2] [3] [4]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- 3D manipulation researchers; RLBench benchmark practitioners; Academic labs working on keyframe visuomotor policies [1] [2]
- Parameters
- Not explicitly stated as a scalar in README or HF model card (rvt2/model_99.pth is 0.87 GB / 0.81 GiB [3] [5]
- Source rights
- Some licenses are unresolved. Includes noncommercial terms. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- License
- NVIDIA Source Code License (non-commercial research/evaluation only) (noncommercial), CC-BY-NC-SA-4.0 (per GitHub README) / NVIDIA Source Code License (per HF model card YAML) (noncommercial) · license not fully assessed for datasets · Details by part
Overview
RVT-2 (Robotic View Transformer 2) is a multi-task, language-conditioned 3D manipulation policy from NVIDIA Research (RSS 2024) released in the backward-compatible NVlabs/RVT repository alongside the original CoRL 2023 RVT model. Both code and pretrained checkpoints are publicly downloadable without gating, but they are restricted to non-commercial research and evaluation. [1] [2] [3] [4]
Model card summary
- Architecture
Two-stage coarse-to-fine multi-view 3D transformer (stage_two: true, depth: 8, attn_dim: 512, attn_heads: 8, img_size: 224, img_patch_size: 14, rend_three_views: true, cvx_up: true, xops: true) using a custom Point-Renderer (no PyTorch3D needed for RVT-2) to render virtual orthographic views from RGB-D point clouds, conditioned on 512-D language embeddings (lang_len: 77) and 4-D proprioception.
- Parameter count
Not explicitly stated as a scalar in README or HF model card (rvt2/model_99.pth is 0.87 GB / 0.81 GiB; rvt/model_14.pth is 0.44 GB / 0.41 GiB)
Checkpoint byte sizes verified directly via Hugging Face API tree metadata (867,812,309 bytes and 436,405,301 bytes).
- Input modalities
- rgbdpoint-cloudproprioceptionlanguage
RGB-D observations lifted to point clouds and re-rendered into 3 virtual views (add_depth: true, add_corr: true, add_proprio: true, proprio_dim: 4, add_lang: true).
- Output modalities
- ee-pose-absgripper-command
Predicts keyframe 3D translation heatmaps, discretized 3D Euler rotations (num_rot: 72), gripper open/close, and collision avoidance flags.
- Action space
Keyframe 6-DoF end-effector target pose (coarse-to-fine 3D translation via multi-view heatmaps + 72-bin discretized rotations per axis) plus binary gripper state and collision flag, executed via motion planner.
- Trained / supported embodiments
- Franka Emika Panda (RLBench simulation and real-robot experiments)
- Training datasets
- PerAct pre-generated 18-task RLBench dataset (100 demos per task)
- Pre-generated YARR RLBench replay buffer (98 GB on Hugging Face datasets/ankgoyal/rvt)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| ankgoyal/rvt (rvt2/model_99.pth) | pytorch | 0.87 GB | open | f4c52270ebd3 |
| ankgoyal/rvt (rvt/model_14.pth) | pytorch | 0.44 GB | open | f4c52270ebd3 |
Ungated Hugging Face downloads, but subject to non-commercial licensing terms.
Is this for you?
- Use cases
- Multi-task language-conditioned 3D manipulation on 18 RLBench tasks
- Few-shot high-precision manipulation (e.g., plug picking and insertion)
- Benchmarking multi-view 3D transformers against PerAct and RVT
- Known limitations
- Non-commercial research and evaluation license only (NVIDIA Source Code License for code; CC-BY-NC-SA-4.0 / NVIDIA Source Code License for weights).
- Tested environment pins legacy Python 3.8, PyTorch 1.12.1, CUDA 11.1/11.3, and CoppeliaSim 4.1.
- Keyframe-based 3D pose prediction relies on a motion planner (e.g., RLBench sampling-based planner) rather than outputting high-frequency continuous joint trajectories.
- Safety and limitations
- Strictly restricted to non-commercial research and evaluation under NVIDIA Source Code License and CC-BY-NC-SA-4.0.
- Relies on calibrated RGB-D point clouds and a downstream motion planner for collision-free execution.
- Evaluation variance on RLBench depends on the underlying sampling-based motion planner.
Compute & fine-tuning
- Inference compute
- Documented hardware: NVIDIA V100 16 GB (single GPU evaluation via --device 0)
- Precision:
fp32 - Edge / consumer capable: No
Requires CUDA GPU for custom Point-Renderer and optional xformers; exact minimum VRAM and latency_ms not stated in README.
[1] · Editorial assessment
- Fine-tuning
- Supported: Yes
- Methods: Full multi-task training via train.py with YAML/opts overrides
- Minimum VRAM: 16 GB
- Recipe: Upstream fine-tuning documentation
README documents training on 8x V100 (16 GB each) in ~1 day for RVT and 6x faster training for RVT-2.
- Host computer
Multi-task RLBench training uses 8x NVIDIA V100 GPUs (16 GB VRAM each); single-GPU evaluation supported via --device 0.
- Operating system
- Linux (Ubuntu 16.04 / 18.04 / 20.04 for CoppeliaSim 4.1)
- Dependencies
- Python 3.8
- PyTorch 1.12.1
- CUDA 11.1 / 11.3
- CoppeliaSim 4.1
- PyRep, RLBench, YARR, peract_colab, point-renderer (submodules)
- xformers (optional, recommended)
- PyTorch3D (only required for RVT v1, skipped for RVT-2)
- Skills
- Python
- PyTorch
- CUDA / xformers
- RLBench / CoppeliaSim
Inferred from documented installation and training/evaluation commands.
[1] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [1]
- Create a Python 3.8 conda environment and install PyTorch 1.12.1 (CUDA 11.3 wheels)
- Download CoppeliaSim 4.1 and export COPPELIASIM_ROOT, LD_LIBRARY_PATH, and QT_QPA_PLATFORM_PLUGIN_PATH
- Clone NVlabs/RVT with --recurse-submodules and install the package (with optional xformers) plus PyRep, RLBench, YARR, peract_colab, and point-renderer
- Download pretrained rvt2/model_99.pth from ankgoyal/rvt and run python eval.py or train.py
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| RLBench 18-task multi-task benchmark (arXiv:2406.08545v1) | 82% average success rate (up from 65% for RVT), 6x faster training and 2x faster inference than RVT | Upstream-reported paper result evaluated on the final checkpoint (model_99.pth) across 5 evaluation runs to account for RLBench sampling-based planner variance. Not independently verified by OSR. |
| Real-robot high-precision manipulation (arXiv:2406.08545v1) | Learns high-precision tasks such as picking up and inserting plugs with 10 demonstrations | Upstream-reported qualitative/quantitative real-world evaluation; not independently verified by OSR. |
Project notes
Two-stage multi-view architecture and Point-Renderer
At the pinned Hugging Face revision (f4c52270ebd3), rvt2/mvt_cfg.yaml defines an 8-layer transformer (attn_dim: 512, attn_heads: 8, img_size: 224, img_patch_size: 14, stage_two: true, xops: true) that projects RGB-D point clouds into three virtual orthographic views (rend_three_views: true) using a custom CUDA Point-Renderer rather than PyTorch3D. Conditioned on 512-D language embeddings (lang_len: 77) and 4-D proprioception, the model predicts coarse-to-fine 3D keyframe translations via multi-view heatmaps together with 72-bin discretized rotations (num_rot: 72) and gripper commands for downstream motion-planner execution. Exact parameter counts are not published in the README or model card; the Hugging Face checkpoints measure approximately 0.87 GB / 0.81 GiB (rvt2/model_99.pth, trained for 99 epochs / ~80K steps) and 0.44 GB / 0.41 GiB (rvt/model_14.pth). [1] [3] [5]
Training footprint, RLBench evaluation route, and upstream-reported benchmarks
Multi-task RLBench training is documented on 8 NVIDIA V100 GPUs (16 GB VRAM each), with a 98 GB pre-generated YARR replay buffer hosted on Hugging Face (datasets/ankgoyal/rvt) as an optional speedup when training from scratch; it is not needed for evaluating pretrained checkpoints. For single-GPU evaluation, the README directs users to place model_99.pth and its config files under RVT/rvt/runs/rvt2/, place the RLBench test data under RVT/rvt/data/test, and run python eval.py --model-folder runs/rvt2 --eval-datafolder ./data/test --tasks all --eval-episodes 25 --log-name test/1 --device 0 --headless --model-name model_99.pth from the RVT/rvt directory. Setup relies on legacy dependencies—Python 3.8, PyTorch 1.12.1 (CUDA 11.1/11.3), and CoppeliaSim 4.1—plus bundled submodules for PyRep, RLBench, YARR, and Point-Renderer. In arXiv:2406.08545v1, the authors report an 82% average success rate across 18 RLBench tasks (up from 65% for RVT), 6x faster training and 2x faster inference than RVT, and real-world high-precision plug picking and insertion from 10 demonstrations; in the README FAQ, the authors hypothesize that RLBench's sampling-based motion planner contributes to evaluation variance, so they evaluate each checkpoint five times and report mean and variance rather than relying on a single run of the example command. All reported scores are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [4]
What we checked
None. Prepared from public upstream sources without hands-on training, inference, or robot execution.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Latest release
- Mar 14, 2025 [3]
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- NVlabs/RVT@367995a1a2169b6352bf4e8b0ed405890462a3a0; HF ankgoyal/rvt@f4c52270ebd3041aba0ebd77f43f4e2c4e660287 · combined hardware and software repository
Separate license matrix
Full source code and pretrained RVT-2 / RVT checkpoints are publicly downloadable, but both code (NVIDIA Source Code License) and weights (CC-BY-NC-SA-4.0 / NVIDIA Source Code License) restrict use to non-commercial research and evaluation.
| Layer | Availability / license |
|---|---|
| Code | available · NVIDIA Source Code License (non-commercial research/evaluation only) Section 3.3 limits use to non-commercial research or evaluation purposes only (excluding NVIDIA Corporation and affiliates). [1] [2] |
| Model weights | available · CC-BY-NC-SA-4.0 (per GitHub README) / NVIDIA Source Code License (per HF model card YAML) Both declared licenses prohibit commercial use; GitHub README states CC-BY-NC-SA-4.0 for pretrained models while HF model card YAML links to NVlabs/RVT LICENSE. [1] [2] [3] |
| Training datasets | available · License not fully assessed Uses PerAct pre-generated RLBench dataset (Google Drive) and 98 GB YARR replay buffer on Hugging Face; separate dataset license is not declared in README. [1] |
| Documentation | available · NVIDIA Source Code License (non-commercial research/evaluation only) [1] [2] |
Non-commercial licensing across code and weights
The NVlabs/RVT repository is governed by the NVIDIA Source Code License (LICENSE at 367995a1a216), whose Section 3.3 restricts the work and derivative works to non-commercial research or evaluation purposes only (excluding NVIDIA Corporation and its affiliates). For pretrained weights, the GitHub README states CC-BY-NC-SA-4.0 while the Hugging Face model card YAML declares nvidia-source-code-license; under either reading, commercial deployment is prohibited. [1] [2] [3]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 NVlabs/RVT GitHub repository and README.md
Documents RVT-2 and RVT installation (Python 3.8, PyTorch 1.12.1, CoppeliaSim 4.1, custom Point-Renderer), 8x V100 training, Hugging Face checkpoints, and CC-BY-NC-SA-4.0 model license note.Revision, date & where cited
NVIDIA Research (GitHub) · github.com · revision
367995a· README.md at commit 367995a1a216 (2025-02-14) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 7 more - S002 NVlabs/RVT LICENSE (NVIDIA Source Code License)
NVIDIA License Section 3.3 restricts the Work and derivative works to non-commercial research or evaluation purposes only.Revision, date & where cited
NVIDIA Corporation · github.com · revision
367995a· LICENSE at commit 367995a1a216 · Accessed Nov 21, 2026
Used for: Overview, Is this for you?, Separate license matrix - S003 Hugging Face model repository: ankgoyal/rvt
Hosts ungated rvt2/model_99.pth (0.87 GB / 0.81 GiB) and rvt/model_14.pth (0.44 GB / 0.41 GiB); YAML metadata declares license: other (nvidia-source-code-license).Revision, date & where cited
Ankit Goyal (Hugging Face) · huggingface.co · HF API & README.md at revision f4c52270ebd3 (2025-03-14) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 4 more - S004 RVT-2: Learning Precise Manipulation from Few Demonstrations (RSS 2024)
Reports 82% average success rate on RLBench (up from 65% for RVT), 6x faster training and 2x faster inference than RVT, and real-world high-precision plug insertion from 10 demos.Revision, date & where cited
arXiv (2406.08545v1) · arxiv.org · arXiv abstract 2406.08545v1 · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Upstream-reported benchmarks and 1 more - S005 RVT-2 model configuration (rvt2/mvt_cfg.yaml)
Confirms stage_two: true, depth: 8, attn_dim: 512, attn_heads: 8, img_size: 224, img_patch_size: 14, rend_three_views: true, use_point_renderer: true, num_rot: 72, lang_dim: 512, proprio_dim: 4.Revision, date & where cited
Ankit Goyal (Hugging Face) · huggingface.co · rvt2/mvt_cfg.yaml at revision f4c52270ebd3 · Accessed Nov 21, 2026
Used for: Model card summary, Two-stage multi-view architecture and Point-Renderer
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile drafted from NVlabs/RVT@367995a1a216 and ankgoyal/rvt@f4c52270ebd3.