Open Source RoboticsRobots, software & learning

VQ-BeT (Vector-Quantized Behavior Transformer)

Robotics model

MIT-licensed multimodal behavior generation policy that tokenizes continuous actions with a hierarchical Residual VQ-VAE and predicts discrete latent action codes plus continuous offsets with a causal GPT transformer; upstream releases training/evaluation code, benchmark datasets, and pretrained goal-conditioned Kitchen weights on Google Drive. [1] [2] [3] [4]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Imitation learning researchers; Robotics behavior cloning practitioners; Offline decision-making and policy-head developers [1] [4]
Parameters
Task-dependent (not published as a single scalar [2]
Embodiments
Franka Kitchen (D4RL simulation), PushT planar manipulation (simulation), Dual/Single UR3 arm (simulation), … [1] [2]
Source rights
Some licenses are unresolved. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware
Ecosystem
MuJoCo
License
MIT · license not fully assessed for model weights, datasets · Details by part

Requirements · Research details and dates

On this page

Overview

VQ-BeT (Vector-Quantized Behavior Transformer) is an MIT-licensed behavior cloning architecture that replaces k-means action discretization in Behavior Transformers (BeT) with hierarchical Residual Vector Quantization. The official repository (jayLEE0301/vq_bet_official) provides two-stage PyTorch training and evaluation scripts for state- and image-based continuous-action tasks, along with Google Drive links to benchmark datasets and pretrained goal-conditioned Kitchen checkpoints. [1] [3] [4]

Model card summary

Architecture

Two-stage Vector-Quantized Behavior Transformer combining a Residual VQ-VAE action tokenizer (n_latent_dims: 512, vqvae_n_embed: 16, vqvae_groups: 2 in Kitchen config) with a causal GPT backbone (n_layer: 6, n_head: 6, n_embd: 120, block_size: 30 in train_kitchen_goalcond.yaml) that predicts discrete primary/secondary latent codes and continuous action offsets; optional frozen ResNet18 visual embeddings for PushT and Kitchen.

[1] [2] [4]

Parameter count

Task-dependent (not published as a single scalar; Kitchen config uses a 6-layer, 6-head, 120-dim GPT + 512-dim Residual VQ-VAE with 2 groups x 16 codes)

Exact parameter count depends on environment observation/action dimensions and GPT/VQ-VAE config.

[2] · Editorial assessment

Input modalities
proprioceptionrgb

Supports state/proprioceptive vectors (e.g., 60-D Kitchen state + optional 60-D goal) and visual observations via pre-processed ResNet18 RGB embeddings (visual_input: true).

[1] [2]

Output modalities
joint-positionsee-pose-delta

Outputs continuous environment-specific actions decoded from Residual VQ codes plus continuous offsets (e.g., 9-D Franka Kitchen actions, 2-D PushT, UR3 arm actions).

[1] [2] [4]

Action space

Continuous action vectors or multi-step action chunks (action_window_size: 1 for Kitchen, 3–5 recommended for correlated tasks like PushT; e.g., 9-D in Kitchen) reconstructed via Residual VQ discrete codebook lookup plus predicted continuous offset.

[1] [2]

Trained / supported embodiments
  • Franka Kitchen (D4RL simulation)
  • PushT planar manipulation (simulation)
  • Dual/Single UR3 arm (simulation)
  • Ant locomotion (simulation)

Paper also evaluates on additional autonomous driving and real-world robotics setups, while this repository ships configs for Kitchen, PushT, UR3, and Ant.

[1] [2]

Training datasets
  • Relay Kitchen (D4RL) dataset
  • PushT, UR3, and Ant datasets bundled in the upstream Google Drive archive

[1] [2]

Interfaces
  • Hydra YAML CLI training scripts (examples/pretrain_vqvae.py, examples/train.py)
  • OpenAI Gym / D4RL environment wrappers
  • PyTorch nn.Module policy API (vq_behavior_transformer.BehaviorTransformer, vqvae.VqVae)

[1] [2]

Checkpoints

CheckpointFormatSizeAccessPinned revision
VQ-BeT + Residual VQ pretrained weights (goal-conditioned Kitchen, Google Drive)pytorchNot statedopenunversioned Google Drive archive (linked from 09d4851288ca)

Archive contains rvq/trained_vqvae.pt and vq-bet checkpoints for goal-conditioned Kitchen; no separate weights license is declared.

[1]

Is this for you?

Use cases
  • Multimodal behavior cloning on continuous-action manipulation tasks
  • Goal-conditioned and unconditional sequential action generation
  • Low-latency alternative to iterative diffusion action heads

[1] [4]

Known limitations
  • Pretrained weights are only provided for goal-conditioned Kitchen; PushT, Ant, UR3, and custom environments require two-stage training (Residual VQ-VAE pretraining followed by VQ-BeT training).
  • Visual observation configs in the pinned repository use frozen pre-processed ResNet18 embeddings; README notes that frozen embeddings can underperform end-to-end ResNet18 fine-tuning.
  • Simulation environments require MuJoCo 2.1.0 / mujoco-py, D4RL, and Cython < 3 on Linux/EGL setups.
  • Linked Google Drive weights and datasets do not declare a separate license.

[1]

Safety and limitations
  • Pretrained weights are only published for goal-conditioned Kitchen; other tasks require training both Residual VQ-VAE and VQ-BeT from scratch.
  • Frozen ResNet18 embeddings for visual inputs can degrade performance relative to end-to-end visual fine-tuning.
  • Repository is a research benchmark implementation without hardware safety interlocks or robot controller drivers.

[1]

Compute & fine-tuning

Inference compute
  • Documented hardware: CUDA GPU (exact model not specified in README)
  • Precision: fp32

Configs default to CUDA fp32; paper reports 5x inference speedup over Diffusion Policies without pinning a single millisecond latency or VRAM floor in the README.

[1] [2] [4] · Editorial assessment

Fine-tuning
  • Supported: Yes
  • Methods: Two-stage per-environment training: Residual VQ-VAE pretraining (examples/pretrain_vqvae.py) followed by VQ-BeT GPT training (examples/train.py)
  • Recipe: Upstream fine-tuning documentation

[1]

Host computer

Single CUDA GPU configured by default (device: cuda, batch_size: 2048 in Kitchen configs); exact GPU model and VRAM floor are not pinned in README.

[1] [2]

Operating system
  • Linux (inferred from MuJoCo/EGL/bashrc instructions)

README documents Linux conda, gcc/glew/patchelf, and MUJOCO_GL=egl workflows.

[1] · Editorial assessment

Dependencies
  • Python 3.7 or 3.9
  • PyTorch (tested on 1.12.1 and 2.1.0)
  • Hydra / OmegaConf
  • MuJoCo 2.1.0 / mujoco-py (with Cython < 3)
  • D4RL (Farama-Foundation/d4rl)
  • WandB (optional)

[1]

Skills
  • Python
  • PyTorch
  • Hydra / WandB
  • MuJoCo / D4RL environment setup

Inferred from README installation, Hydra YAML configuration, and MuJoCo troubleshooting notes.

[1] [2] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1] · Model card / documentation [1]

  • Create a Python 3.7 or 3.9 conda environment and install PyTorch (tested on 1.12.1 and 2.1.0)
  • Install package requirements (pip install -r requirements.txt or sh install.sh) and editable package (pip install -e .)
  • Install MuJoCo, Farama-Foundation/d4rl, and optionally envs/ur3 in editable mode
  • Download datasets and optional pretrained Kitchen weights from Google Drive, set paths in examples/configs/env_vars/env_vars.yaml, then run examples/pretrain_vqvae.py and examples/train.py

[1]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
Seven behavior-generation environments including simulated manipulation, autonomous driving, and robotics (arXiv:2403.03181v2)Improves over BeT and Diffusion Policies across evaluated tasks while accelerating inference speed 5x over Diffusion PoliciesUpstream-reported paper abstract result across 7 environments; this repository provides configs for Kitchen, PushT, UR3, and Ant. Not independently verified by OSR.

[4]

Project notes

Residual VQ tokenization and GPT policy backbone

At the pinned commit (09d4851288ca), VQ-BeT separates action modeling into two stages. First, examples/pretrain_vqvae.py trains a Residual VQ-VAE (vqvae.VqVae, configured in pretrain_kitchen.yaml with n_latent_dims: 512, vqvae_n_embed: 16, and vqvae_groups: 2) to map continuous single-step actions or multi-step action chunks (action_window_size) into discrete latent codes. Second, examples/train.py trains a causal GPT backbone (vq_behavior_transformer.GPT, configured in train_kitchen_goalcond.yaml with n_layer: 6, n_head: 6, n_embd: 120, and block_size: 30) that predicts discrete primary and secondary action codes together with continuous offsets. Exact parameter counts are not stated as a single scalar and depend on environment state/action dimensions and Hydra config choices. [1] [2]

Setup constraints and upstream-reported results

The repository ships Hydra configs for Franka Kitchen, PushT, UR3, and Ant in both goal-conditioned and unconditional variants, plus template configs for custom OpenAI Gym environments. Installation targets Python 3.7 or 3.9, PyTorch 1.12.1 or 2.1.0, MuJoCo 2.1.0 (mujoco-py with cython<3), and D4RL. The supplied Kitchen configs default to device: cuda and batch_size: 2048, though those are configuration defaults rather than measured hardware floors; the reviewed materials do not establish a minimum GPU model or VRAM requirement. For visual observation tasks (visual_input: true), the pinned repository relies on frozen pre-processed ResNet18 embeddings for PushT and Kitchen; upstream notes that frozen embeddings can yield lower performance than fine-tuning ResNet18 end-to-end. In arXiv:2403.03181v2, the authors report improvements over BeT and Diffusion Policies across seven environments spanning simulated manipulation, autonomous driving, and robotics, alongside a 5x inference speedup over Diffusion Policies. All reported metrics are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [2] [4]

What we checked

None. Prepared from public upstream sources without hands-on training, inference, or robot execution.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Feb 28, 2024 [1]
Latest release
Not established
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
jayLEE0301/vq_bet_official@09d4851288ca5deaaa1ab367a208e520f8ee9a84 · software repository

Separate license matrix

MIT-licensed multimodal behavior cloning codebase with downloadable datasets and goal-conditioned Kitchen checkpoints, included as partial because the Google Drive weights and datasets lack an explicit separate license and only Kitchen weights are pre-released.

LayerAvailability / license
Codeavailable · MIT
Root LICENSE is MIT (Copyright 2024 Seungjae Lee); bundled UR3 driver code (envs/ur3/.../urx/COPYING) retains its own upstream license. [1] [3]
Model weightsavailable · License not fully assessed
Pretrained Residual VQ and VQ-BeT weights for goal-conditioned Kitchen are hosted on Google Drive without a separately declared weights license. [1]
Training datasetsavailable · License not fully assessed
Benchmark datasets are hosted on Google Drive without a separately declared dataset license. [1]
Documentationavailable · MIT
[1] [3]

Licensing and weight availability boundaries

The repository codebase and documentation are licensed under the MIT License (LICENSE at 09d4851288ca, Copyright 2024 Seungjae Lee), while bundled UR3 driver code retains its upstream license notice. Pretrained Residual VQ and VQ-BeT checkpoints are published via Google Drive only for goal-conditioned Kitchen, and neither the linked weights archive nor the linked dataset archive declares a separate license. [1] [3]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile drafted from jayLEE0301/vq_bet_official@09d4851288ca.