Open Source RoboticsRobots, software & learning

RVT-2 (Robotic View Transformer 2)

Robotics model

Two-stage coarse-to-fine multi-view 3D transformer policy from NVIDIA Research (RSS 2024) for language-conditioned, high-precision keyframe manipulation on RLBench and real robots, with source-available non-commercial code and open-download non-commercial checkpoints on Hugging Face. [1] [2] [3] [4]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
3D manipulation researchers; RLBench benchmark practitioners; Academic labs working on keyframe visuomotor policies [1] [2]
Parameters
Not explicitly stated as a scalar in README or HF model card (rvt2/model_99.pth is 0.87 GB / 0.81 GiB [3] [5]
Embodiments
Franka Emika Panda (RLBench simulation and real-robot experiments) [1] [4]
Source rights
Some licenses are unresolved. Includes noncommercial terms. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware

Requirements · Research details and dates

On this page

Overview

RVT-2 (Robotic View Transformer 2) is a multi-task, language-conditioned 3D manipulation policy from NVIDIA Research (RSS 2024) released in the backward-compatible NVlabs/RVT repository alongside the original CoRL 2023 RVT model. Both code and pretrained checkpoints are publicly downloadable without gating, but they are restricted to non-commercial research and evaluation. [1] [2] [3] [4]

Model card summary

Architecture

Two-stage coarse-to-fine multi-view 3D transformer (stage_two: true, depth: 8, attn_dim: 512, attn_heads: 8, img_size: 224, img_patch_size: 14, rend_three_views: true, cvx_up: true, xops: true) using a custom Point-Renderer (no PyTorch3D needed for RVT-2) to render virtual orthographic views from RGB-D point clouds, conditioned on 512-D language embeddings (lang_len: 77) and 4-D proprioception.

[1] [5]

Parameter count

Not explicitly stated as a scalar in README or HF model card (rvt2/model_99.pth is 0.87 GB / 0.81 GiB; rvt/model_14.pth is 0.44 GB / 0.41 GiB)

Checkpoint byte sizes verified directly via Hugging Face API tree metadata (867,812,309 bytes and 436,405,301 bytes).

[3] [5] · Editorial assessment

Input modalities
rgbdpoint-cloudproprioceptionlanguage

RGB-D observations lifted to point clouds and re-rendered into 3 virtual views (add_depth: true, add_corr: true, add_proprio: true, proprio_dim: 4, add_lang: true).

[1] [5]

Output modalities
ee-pose-absgripper-command

Predicts keyframe 3D translation heatmaps, discretized 3D Euler rotations (num_rot: 72), gripper open/close, and collision avoidance flags.

[1] [5]

Action space

Keyframe 6-DoF end-effector target pose (coarse-to-fine 3D translation via multi-view heatmaps + 72-bin discretized rotations per axis) plus binary gripper state and collision flag, executed via motion planner.

[1] [5]

Trained / supported embodiments
  • Franka Emika Panda (RLBench simulation and real-robot experiments)

[1] [4]

Training datasets
  • PerAct pre-generated 18-task RLBench dataset (100 demos per task)
  • Pre-generated YARR RLBench replay buffer (98 GB on Hugging Face datasets/ankgoyal/rvt)

[1]

Interfaces
  • Python CLI (train.py, eval.py)
  • YAML config + CLI opts overrides
  • RLBench / YARR environment interface

[1] [5]

Checkpoints

CheckpointFormatSizeAccessPinned revision
ankgoyal/rvt (rvt2/model_99.pth)pytorch0.87 GBopenf4c52270ebd3
ankgoyal/rvt (rvt/model_14.pth)pytorch0.44 GBopenf4c52270ebd3

Ungated Hugging Face downloads, but subject to non-commercial licensing terms.

[1] [3]

Is this for you?

Use cases
  • Multi-task language-conditioned 3D manipulation on 18 RLBench tasks
  • Few-shot high-precision manipulation (e.g., plug picking and insertion)
  • Benchmarking multi-view 3D transformers against PerAct and RVT

[1] [4]

Known limitations
  • Non-commercial research and evaluation license only (NVIDIA Source Code License for code; CC-BY-NC-SA-4.0 / NVIDIA Source Code License for weights).
  • Tested environment pins legacy Python 3.8, PyTorch 1.12.1, CUDA 11.1/11.3, and CoppeliaSim 4.1.
  • Keyframe-based 3D pose prediction relies on a motion planner (e.g., RLBench sampling-based planner) rather than outputting high-frequency continuous joint trajectories.

[1] [2] [3]

Safety and limitations
  • Strictly restricted to non-commercial research and evaluation under NVIDIA Source Code License and CC-BY-NC-SA-4.0.
  • Relies on calibrated RGB-D point clouds and a downstream motion planner for collision-free execution.
  • Evaluation variance on RLBench depends on the underlying sampling-based motion planner.

[1] [2] [3]

Compute & fine-tuning

Inference compute
  • Documented hardware: NVIDIA V100 16 GB (single GPU evaluation via --device 0)
  • Precision: fp32
  • Edge / consumer capable: No

Requires CUDA GPU for custom Point-Renderer and optional xformers; exact minimum VRAM and latency_ms not stated in README.

[1] · Editorial assessment

Fine-tuning

README documents training on 8x V100 (16 GB each) in ~1 day for RVT and 6x faster training for RVT-2.

[1]

Host computer

Multi-task RLBench training uses 8x NVIDIA V100 GPUs (16 GB VRAM each); single-GPU evaluation supported via --device 0.

[1]

Operating system
  • Linux (Ubuntu 16.04 / 18.04 / 20.04 for CoppeliaSim 4.1)

[1]

Dependencies
  • Python 3.8
  • PyTorch 1.12.1
  • CUDA 11.1 / 11.3
  • CoppeliaSim 4.1
  • PyRep, RLBench, YARR, peract_colab, point-renderer (submodules)
  • xformers (optional, recommended)
  • PyTorch3D (only required for RVT v1, skipped for RVT-2)

[1]

Skills
  • Python
  • PyTorch
  • CUDA / xformers
  • RLBench / CoppeliaSim

Inferred from documented installation and training/evaluation commands.

[1] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1] · Model card / documentation [1]

  • Create a Python 3.8 conda environment and install PyTorch 1.12.1 (CUDA 11.3 wheels)
  • Download CoppeliaSim 4.1 and export COPPELIASIM_ROOT, LD_LIBRARY_PATH, and QT_QPA_PLATFORM_PLUGIN_PATH
  • Clone NVlabs/RVT with --recurse-submodules and install the package (with optional xformers) plus PyRep, RLBench, YARR, peract_colab, and point-renderer
  • Download pretrained rvt2/model_99.pth from ankgoyal/rvt and run python eval.py or train.py

[1] [3]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
RLBench 18-task multi-task benchmark (arXiv:2406.08545v1)82% average success rate (up from 65% for RVT), 6x faster training and 2x faster inference than RVTUpstream-reported paper result evaluated on the final checkpoint (model_99.pth) across 5 evaluation runs to account for RLBench sampling-based planner variance. Not independently verified by OSR.
Real-robot high-precision manipulation (arXiv:2406.08545v1)Learns high-precision tasks such as picking up and inserting plugs with 10 demonstrationsUpstream-reported qualitative/quantitative real-world evaluation; not independently verified by OSR.

[1] [4]

Project notes

Two-stage multi-view architecture and Point-Renderer

At the pinned Hugging Face revision (f4c52270ebd3), rvt2/mvt_cfg.yaml defines an 8-layer transformer (attn_dim: 512, attn_heads: 8, img_size: 224, img_patch_size: 14, stage_two: true, xops: true) that projects RGB-D point clouds into three virtual orthographic views (rend_three_views: true) using a custom CUDA Point-Renderer rather than PyTorch3D. Conditioned on 512-D language embeddings (lang_len: 77) and 4-D proprioception, the model predicts coarse-to-fine 3D keyframe translations via multi-view heatmaps together with 72-bin discretized rotations (num_rot: 72) and gripper commands for downstream motion-planner execution. Exact parameter counts are not published in the README or model card; the Hugging Face checkpoints measure approximately 0.87 GB / 0.81 GiB (rvt2/model_99.pth, trained for 99 epochs / ~80K steps) and 0.44 GB / 0.41 GiB (rvt/model_14.pth). [1] [3] [5]

Training footprint, RLBench evaluation route, and upstream-reported benchmarks

Multi-task RLBench training is documented on 8 NVIDIA V100 GPUs (16 GB VRAM each), with a 98 GB pre-generated YARR replay buffer hosted on Hugging Face (datasets/ankgoyal/rvt) as an optional speedup when training from scratch; it is not needed for evaluating pretrained checkpoints. For single-GPU evaluation, the README directs users to place model_99.pth and its config files under RVT/rvt/runs/rvt2/, place the RLBench test data under RVT/rvt/data/test, and run python eval.py --model-folder runs/rvt2 --eval-datafolder ./data/test --tasks all --eval-episodes 25 --log-name test/1 --device 0 --headless --model-name model_99.pth from the RVT/rvt directory. Setup relies on legacy dependencies—Python 3.8, PyTorch 1.12.1 (CUDA 11.1/11.3), and CoppeliaSim 4.1—plus bundled submodules for PyRep, RLBench, YARR, and Point-Renderer. In arXiv:2406.08545v1, the authors report an 82% average success rate across 18 RLBench tasks (up from 65% for RVT), 6x faster training and 2x faster inference than RVT, and real-world high-precision plug picking and insertion from 10 demonstrations; in the README FAQ, the authors hypothesize that RLBench's sampling-based motion planner contributes to evaluation variance, so they evaluate each checkpoint five times and report mean and variance rather than relying on a single run of the example command. All reported scores are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [4]

What we checked

None. Prepared from public upstream sources without hands-on training, inference, or robot execution.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Mar 14, 2025 [1] [3]
Latest release
Mar 14, 2025 [3]
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
NVlabs/RVT@367995a1a2169b6352bf4e8b0ed405890462a3a0; HF ankgoyal/rvt@f4c52270ebd3041aba0ebd77f43f4e2c4e660287 · combined hardware and software repository

Separate license matrix

Full source code and pretrained RVT-2 / RVT checkpoints are publicly downloadable, but both code (NVIDIA Source Code License) and weights (CC-BY-NC-SA-4.0 / NVIDIA Source Code License) restrict use to non-commercial research and evaluation.

LayerAvailability / license
Codeavailable · NVIDIA Source Code License (non-commercial research/evaluation only)
Section 3.3 limits use to non-commercial research or evaluation purposes only (excluding NVIDIA Corporation and affiliates). [1] [2]
Model weightsavailable · CC-BY-NC-SA-4.0 (per GitHub README) / NVIDIA Source Code License (per HF model card YAML)
Both declared licenses prohibit commercial use; GitHub README states CC-BY-NC-SA-4.0 for pretrained models while HF model card YAML links to NVlabs/RVT LICENSE. [1] [2] [3]
Training datasetsavailable · License not fully assessed
Uses PerAct pre-generated RLBench dataset (Google Drive) and 98 GB YARR replay buffer on Hugging Face; separate dataset license is not declared in README. [1]
Documentationavailable · NVIDIA Source Code License (non-commercial research/evaluation only)
[1] [2]

Non-commercial licensing across code and weights

The NVlabs/RVT repository is governed by the NVIDIA Source Code License (LICENSE at 367995a1a216), whose Section 3.3 restricts the work and derivative works to non-commercial research or evaluation purposes only (excluding NVIDIA Corporation and its affiliates). For pretrained weights, the GitHub README states CC-BY-NC-SA-4.0 while the Hugging Face model card YAML declares nvidia-source-code-license; under either reading, commercial deployment is prohibited. [1] [2] [3]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile drafted from NVlabs/RVT@367995a1a216 and ankgoyal/rvt@f4c52270ebd3.