VQ-BeT (Vector-Quantized Behavior Transformer)
Robotics modelMIT-licensed multimodal behavior generation policy that tokenizes continuous actions with a hierarchical Residual VQ-VAE and predicts discrete latent action codes plus continuous offsets with a causal GPT transformer; upstream releases training/evaluation code, benchmark datasets, and pretrained goal-conditioned Kitchen weights on Google Drive. [1] [2] [3] [4]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Imitation learning researchers; Robotics behavior cloning practitioners; Offline decision-making and policy-head developers [1] [4]
- Parameters
- Task-dependent (not published as a single scalar [2]
- Embodiments
- Franka Kitchen (D4RL simulation), PushT planar manipulation (simulation), Dual/Single UR3 arm (simulation), … [1] [2]
- Source rights
- Some licenses are unresolved. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- Ecosystem
- MuJoCo
- License
- MIT · license not fully assessed for model weights, datasets · Details by part
Overview
VQ-BeT (Vector-Quantized Behavior Transformer) is an MIT-licensed behavior cloning architecture that replaces k-means action discretization in Behavior Transformers (BeT) with hierarchical Residual Vector Quantization. The official repository (jayLEE0301/vq_bet_official) provides two-stage PyTorch training and evaluation scripts for state- and image-based continuous-action tasks, along with Google Drive links to benchmark datasets and pretrained goal-conditioned Kitchen checkpoints. [1] [3] [4]
Model card summary
- Architecture
Two-stage Vector-Quantized Behavior Transformer combining a Residual VQ-VAE action tokenizer (n_latent_dims: 512, vqvae_n_embed: 16, vqvae_groups: 2 in Kitchen config) with a causal GPT backbone (n_layer: 6, n_head: 6, n_embd: 120, block_size: 30 in train_kitchen_goalcond.yaml) that predicts discrete primary/secondary latent codes and continuous action offsets; optional frozen ResNet18 visual embeddings for PushT and Kitchen.
- Parameter count
Task-dependent (not published as a single scalar; Kitchen config uses a 6-layer, 6-head, 120-dim GPT + 512-dim Residual VQ-VAE with 2 groups x 16 codes)
Exact parameter count depends on environment observation/action dimensions and GPT/VQ-VAE config.
[2] · Editorial assessment
- Input modalities
- proprioceptionrgb
Supports state/proprioceptive vectors (e.g., 60-D Kitchen state + optional 60-D goal) and visual observations via pre-processed ResNet18 RGB embeddings (visual_input: true).
- Output modalities
- joint-positionsee-pose-delta
Outputs continuous environment-specific actions decoded from Residual VQ codes plus continuous offsets (e.g., 9-D Franka Kitchen actions, 2-D PushT, UR3 arm actions).
- Action space
Continuous action vectors or multi-step action chunks (action_window_size: 1 for Kitchen, 3–5 recommended for correlated tasks like PushT; e.g., 9-D in Kitchen) reconstructed via Residual VQ discrete codebook lookup plus predicted continuous offset.
- Trained / supported embodiments
- Franka Kitchen (D4RL simulation)
- PushT planar manipulation (simulation)
- Dual/Single UR3 arm (simulation)
- Ant locomotion (simulation)
Paper also evaluates on additional autonomous driving and real-world robotics setups, while this repository ships configs for Kitchen, PushT, UR3, and Ant.
- Training datasets
- Relay Kitchen (D4RL) dataset
- PushT, UR3, and Ant datasets bundled in the upstream Google Drive archive
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| VQ-BeT + Residual VQ pretrained weights (goal-conditioned Kitchen, Google Drive) | pytorch | Not stated | open | unversioned Google Drive archive (linked from 09d4851288ca) |
Archive contains rvq/trained_vqvae.pt and vq-bet checkpoints for goal-conditioned Kitchen; no separate weights license is declared.
Is this for you?
- Use cases
- Multimodal behavior cloning on continuous-action manipulation tasks
- Goal-conditioned and unconditional sequential action generation
- Low-latency alternative to iterative diffusion action heads
- Known limitations
- Pretrained weights are only provided for goal-conditioned Kitchen; PushT, Ant, UR3, and custom environments require two-stage training (Residual VQ-VAE pretraining followed by VQ-BeT training).
- Visual observation configs in the pinned repository use frozen pre-processed ResNet18 embeddings; README notes that frozen embeddings can underperform end-to-end ResNet18 fine-tuning.
- Simulation environments require MuJoCo 2.1.0 / mujoco-py, D4RL, and Cython < 3 on Linux/EGL setups.
- Linked Google Drive weights and datasets do not declare a separate license.
- Safety and limitations
- Pretrained weights are only published for goal-conditioned Kitchen; other tasks require training both Residual VQ-VAE and VQ-BeT from scratch.
- Frozen ResNet18 embeddings for visual inputs can degrade performance relative to end-to-end visual fine-tuning.
- Repository is a research benchmark implementation without hardware safety interlocks or robot controller drivers.
Compute & fine-tuning
- Inference compute
- Documented hardware: CUDA GPU (exact model not specified in README)
- Precision:
fp32
Configs default to CUDA fp32; paper reports 5x inference speedup over Diffusion Policies without pinning a single millisecond latency or VRAM floor in the README.
- Fine-tuning
- Supported: Yes
- Methods: Two-stage per-environment training: Residual VQ-VAE pretraining (examples/pretrain_vqvae.py) followed by VQ-BeT GPT training (examples/train.py)
- Recipe: Upstream fine-tuning documentation
- Host computer
Single CUDA GPU configured by default (device: cuda, batch_size: 2048 in Kitchen configs); exact GPU model and VRAM floor are not pinned in README.
- Operating system
- Linux (inferred from MuJoCo/EGL/bashrc instructions)
README documents Linux conda, gcc/glew/patchelf, and MUJOCO_GL=egl workflows.
[1] · Editorial assessment
- Dependencies
- Python 3.7 or 3.9
- PyTorch (tested on 1.12.1 and 2.1.0)
- Hydra / OmegaConf
- MuJoCo 2.1.0 / mujoco-py (with Cython < 3)
- D4RL (Farama-Foundation/d4rl)
- WandB (optional)
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [1]
- Create a Python 3.7 or 3.9 conda environment and install PyTorch (tested on 1.12.1 and 2.1.0)
- Install package requirements (pip install -r requirements.txt or sh install.sh) and editable package (pip install -e .)
- Install MuJoCo, Farama-Foundation/d4rl, and optionally envs/ur3 in editable mode
- Download datasets and optional pretrained Kitchen weights from Google Drive, set paths in examples/configs/env_vars/env_vars.yaml, then run examples/pretrain_vqvae.py and examples/train.py
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| Seven behavior-generation environments including simulated manipulation, autonomous driving, and robotics (arXiv:2403.03181v2) | Improves over BeT and Diffusion Policies across evaluated tasks while accelerating inference speed 5x over Diffusion Policies | Upstream-reported paper abstract result across 7 environments; this repository provides configs for Kitchen, PushT, UR3, and Ant. Not independently verified by OSR. |
Project notes
Residual VQ tokenization and GPT policy backbone
At the pinned commit (09d4851288ca), VQ-BeT separates action modeling into two stages. First, examples/pretrain_vqvae.py trains a Residual VQ-VAE (vqvae.VqVae, configured in pretrain_kitchen.yaml with n_latent_dims: 512, vqvae_n_embed: 16, and vqvae_groups: 2) to map continuous single-step actions or multi-step action chunks (action_window_size) into discrete latent codes. Second, examples/train.py trains a causal GPT backbone (vq_behavior_transformer.GPT, configured in train_kitchen_goalcond.yaml with n_layer: 6, n_head: 6, n_embd: 120, and block_size: 30) that predicts discrete primary and secondary action codes together with continuous offsets. Exact parameter counts are not stated as a single scalar and depend on environment state/action dimensions and Hydra config choices. [1] [2]
Setup constraints and upstream-reported results
The repository ships Hydra configs for Franka Kitchen, PushT, UR3, and Ant in both goal-conditioned and unconditional variants, plus template configs for custom OpenAI Gym environments. Installation targets Python 3.7 or 3.9, PyTorch 1.12.1 or 2.1.0, MuJoCo 2.1.0 (mujoco-py with cython<3), and D4RL. The supplied Kitchen configs default to device: cuda and batch_size: 2048, though those are configuration defaults rather than measured hardware floors; the reviewed materials do not establish a minimum GPU model or VRAM requirement. For visual observation tasks (visual_input: true), the pinned repository relies on frozen pre-processed ResNet18 embeddings for PushT and Kitchen; upstream notes that frozen embeddings can yield lower performance than fine-tuning ResNet18 end-to-end. In arXiv:2403.03181v2, the authors report improvements over BeT and Diffusion Policies across seven environments spanning simulated manipulation, autonomous driving, and robotics, alongside a 5x inference speedup over Diffusion Policies. All reported metrics are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [2] [4]
What we checked
None. Prepared from public upstream sources without hands-on training, inference, or robot execution.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Upstream last updated
- Feb 28, 2024 [1]
- Latest release
- Not established
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- jayLEE0301/vq_bet_official@09d4851288ca5deaaa1ab367a208e520f8ee9a84 · software repository
Separate license matrix
MIT-licensed multimodal behavior cloning codebase with downloadable datasets and goal-conditioned Kitchen checkpoints, included as partial because the Google Drive weights and datasets lack an explicit separate license and only Kitchen weights are pre-released.
| Layer | Availability / license |
|---|---|
| Code | available · MIT Root LICENSE is MIT (Copyright 2024 Seungjae Lee); bundled UR3 driver code (envs/ur3/.../urx/COPYING) retains its own upstream license. [1] [3] |
| Model weights | available · License not fully assessed Pretrained Residual VQ and VQ-BeT weights for goal-conditioned Kitchen are hosted on Google Drive without a separately declared weights license. [1] |
| Training datasets | available · License not fully assessed Benchmark datasets are hosted on Google Drive without a separately declared dataset license. [1] |
| Documentation | available · MIT [1] [3] |
Licensing and weight availability boundaries
The repository codebase and documentation are licensed under the MIT License (LICENSE at 09d4851288ca, Copyright 2024 Seungjae Lee), while bundled UR3 driver code retains its upstream license notice. Pretrained Residual VQ and VQ-BeT checkpoints are published via Google Drive only for goal-conditioned Kitchen, and neither the linked weights archive nor the linked dataset archive declares a separate license. [1] [3]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 jayLEE0301/vq_bet_official GitHub repository and README.md
Documents installation (Python 3.7/3.9, PyTorch 1.12.1/2.1.0, MuJoCo, D4RL), two-stage Residual VQ + VQ-BeT training, frozen ResNet18 visual input caveat, and Google Drive links for datasets and pretrained goal-conditioned Kitchen weights.Revision, date & where cited
Seungjae Lee (GitHub) · github.com · revision
09d4851· README.md at commit 09d4851288ca (2024-02-28) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 6 more - S002 VQ-BeT Kitchen Hydra configurations (pretrain_kitchen.yaml and train_kitchen_goalcond.yaml)
Specifies GPT backbone (n_layer: 6, n_head: 6, n_embd: 120, block_size: 30), Residual VQ-VAE (n_latent_dims: 512, vqvae_n_embed: 16, vqvae_groups: 2), obs_dim: 60, act_dim: 9, window_size: 10, action_window_size: 1, batch_size: 2048.Revision, date & where cited
Seungjae Lee (GitHub) · github.com · revision
09d4851· examples/configs/train_kitchen_goalcond.yaml at commit 09d4851288ca · Accessed Nov 21, 2026
Used for: Model card summary, Compute & fine-tuning, Residual VQ tokenization and GPT policy backbone, Setup constraints and upstream-reported results - S003 jayLEE0301/vq_bet_official LICENSE
MIT License, Copyright (c) 2024 Seungjae Lee.Revision, date & where cited
Seungjae Lee (GitHub) · github.com · revision
09d4851· LICENSE at commit 09d4851288ca · Accessed Nov 21, 2026
Used for: Overview, Separate license matrix - S004 Behavior Generation with Latent Actions (arXiv:2403.03181v2)
Introduces VQ-BeT using hierarchical vector quantization of continuous actions; reports improvements over BeT and Diffusion Policies across 7 environments with 5x faster inference than Diffusion Policies.Revision, date & where cited
arXiv (2403.03181v2) · arxiv.org · arXiv abstract 2403.03181v2 · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Compute & fine-tuning and 2 more
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile drafted from jayLEE0301/vq_bet_official@09d4851288ca.