Open Source RoboticsRobots, software & learning

iVideoGPT

Robotics model

MIT-licensed autoregressive transformer world model (NeurIPS 2024, Tsinghua THUML) combining a compressive context-aware VQ-GAN tokenizer (114M/310M) and Llama-style GPT (138M/436M), pre-trained on Open X-Embodiment with 9 ungated Hugging Face checkpoints. [1] [2] [3] [4]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Model-based reinforcement learning researchers; Robotics world-model and video-prediction researchers; Open-weights robotics practitioners seeking lightweight MIT checkpoints [2] [3] [4]
Parameters
252M to 550M total (114M tokenizer + 138M transformer = 252M [2]
Embodiments
Open X-Embodiment multi-robot manipulation datasets (action-free and goal-conditioned pre-training), BAIR robot… [2]
Source rights
Some licenses are unresolved. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware
License
MIT · license not fully assessed for datasets · Details by part

Requirements · Research details and dates

On this page

Overview

iVideoGPT (NeurIPS 2024, Tsinghua THUML) is a scalable autoregressive Transformer world model that interleaves visual, action, and reward tokens for next-token prediction. Its architecture pairs a compressive context-aware VQ-GAN tokenizer (ctx_vqgan, 114M parameters at 64x64 or 310M at 256x256) with a Llama-architecture GPT backbone (138M or 436M parameters), yielding 252M–550M total parameters across the released variants. The thuml/iVideoGPT repository provides pre-training, two-stage fine-tuning, evaluation, visual planning, and Meta-World model-based RL (MBPO) code under the MIT License. [1] [2] [4]

Model card summary

Architecture

Autoregressive Transformer (Llama-style GPT, 138M or 436M) paired with a compressive context-aware VQ-GAN video tokenizer (ctx_vqgan, 114M at 64x64 or 310M at 256x256) that interleaves visual, action, and reward tokens for next-token world-model prediction.

[2] [4]

Base model

Trained from scratch (ctx_vqgan tokenizer + Llama-architecture GPT transformer)

[2]

Parameter count

252M to 550M total (114M tokenizer + 138M transformer = 252M; 114M + 436M = 550M medium; 310M + 138M = 448M at 256x256)

[2]

Input modalities
rgbproprioception

Visual RGB frames (64x64 or 256x256), plus optional low-dimensional robot actions, rewards, or goal images in conditioned variants.

[2] [4]

Output modalities
video-frames

Generates future RGB video frames (64x64 or 256x256) and reward tokens in MBRL mode.

[2] [4]

Action space

Action-free or goal-conditioned on Open X-Embodiment; environment-specific low-dimensional continuous actions (e.g., --action_dim 4 for BAIR) in fine-tuned action-conditioned checkpoints, visual planning, and Meta-World MBRL.

[2] [4]

Control frequency

Not specified (operates on discrete dataset step horizons, e.g., segment_length 8-16 and video_stepsize 1)

[2]

Trained / supported embodiments
  • Open X-Embodiment multi-robot manipulation datasets (action-free and goal-conditioned pre-training)
  • BAIR robot pushing and RoboNet multi-robot arms
  • RoboSuite, RoboDesk, and Meta-World simulated manipulation benchmarks

[2]

Training datasets
  • Open X-Embodiment (OXE_SELECT subset) for pre-training
  • BAIR Robot Pushing, RoboNet, RoboSuite, RoboDesk, and Meta-World for downstream fine-tuning and control

[2]

Interfaces
  • Python CLI (inference/predict.py, train_tokenizer.py, train_gpt.py)
  • Hugging Face Hub safetensors checkpoints
  • MBRL / visual planning scripts (mbrl/train_metaworld_mbpo.py, vp/)

[2] [3]

Checkpoints

CheckpointFormatSizeAccessPinned revision
thuml/ivideogpt-oxe-64-act-freesafetensors1.01 GBopendbca9c7c335f
thuml/ivideogpt-oxe-64-act-free-mediumsafetensors2.2 GBopen0076304ff769
thuml/ivideogpt-oxe-64-goal-condsafetensors1.01 GBopen4d8dfae1e4cb
thuml/ivideogpt-oxe-256-act-freesafetensors1.8 GBopen675bfd148276
thuml/ivideogpt-bair-64-act-freesafetensors1.01 GBopen93e00a1f9981
thuml/ivideogpt-bair-64-act-condsafetensors1.01 GBopen2863382adad1
thuml/ivideogpt-robonet-64-act-condsafetensors1.01 GBopeneb88e1481f49
thuml/ivideogpt-vp2-robosuite-64-act-condsafetensors1.01 GBopendfe4c1db8777
thuml/ivideogpt-vp2-robodesk-64-act-condsafetensors1.01 GBopen6f974cfbee8f

Sizes are decimal GB sums of sibling safetensors/config files at each pinned HF revision. All 9 checkpoints are ungated and MIT-licensed.

[2] [3]

Is this for you?

Use cases
  • Action-free and goal-conditioned video prediction pre-trained on Open X-Embodiment
  • Fine-tuned action-conditioned video prediction (BAIR, RoboNet, RoboSuite, RoboDesk)
  • Visual model-based RL (MBPO on Meta-World) and visual planning

[2] [4]

Known limitations
  • No action-conditioned Open X-Embodiment pre-trained checkpoint is provided due to heterogeneous action spaces across OXE datasets
  • Released 256x256 OXE checkpoint is explicitly noted by upstream as under-trained (intended mainly as a fine-tuning starting point); most released checkpoints operate at 64x64 resolution
  • RoboNet 256x256 checkpoints were accidentally deleted during a disk cleanup and are not available
  • Minimum GPU VRAM requirements for pre-training, fine-tuning, and inference are not explicitly stated in the README

[2]

Safety and limitations
  • Outputs predicted low-resolution video frames (primarily 64x64) and imagined rollouts for planning/MBRL rather than safety-certified physical robot control.
  • No action-conditioned OXE pre-trained model is available; users must fine-tune on their target action space.
  • 256x256 OXE checkpoint is explicitly flagged upstream as insufficiently trained for best standalone performance, and RoboNet 256x256 checkpoints were lost during cleanup.
  • Minimum VRAM and inference latency figures are not reported in the repository README.

[2]

Compute & fine-tuning

Inference compute
  • Documented hardware: Not specified in README (bf16 accelerate training documented; 1.01-2.20 GB checkpoint footprints)
  • Precision: bf16
  • Edge / consumer capable: No

Exact minimum GPU VRAM and inference latency are not stated in the README.

[2] [3] · Editorial assessment

Fine-tuning
  • Supported: Yes
  • Methods: Two-stage fine-tuning (ctx_vqgan tokenizer via train_tokenizer.py + Llama transformer via train_gpt.py); Visual model-based RL (MBPO on Meta-World)
  • Recipe: Upstream fine-tuning documentation

[2]

Host computer

GPU-accelerated Python 3.9 environment (exact minimum VRAM not stated upstream; bf16 mixed precision used in documented accelerate fine-tuning commands; checkpoints are 1.01-2.20 GB safetensors).

[2] [3]

Operating system
  • Linux

Inferred from shell scripts, Conda/CUDA workflow, and Linux MBRL paths.

[2] · Editorial assessment

Dependencies
  • Python 3.9
  • PyTorch and Hugging Face Accelerate
  • Hugging Face Diffusers (codebase base)
  • DrQ-v2 / Meta-World (pinned commit 83ac03ca for MBRL)
  • Pretrained I3D TorchScript model (optional for FVD evaluation)

[2]

Skills
  • Python 3.9 and Conda environment management
  • PyTorch, Hugging Face Diffusers, and Accelerate
  • RL / Meta-World setup for MBRL experiments

Inferred from README installation, accelerate launch commands, and MBRL setup.

[2] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [2] · Model card / documentation [1] [2]

  • Create a Python 3.9 Conda environment and install dependencies from requirements.txt
  • Run inference/predict.py pointing to any ungated Hugging Face checkpoint (e.g., thuml/ivideogpt-oxe-64-act-free)
  • For training or fine-tuning, convert Open X-Embodiment or benchmark datasets with datasets/oxe_data_converter.py and launch train_tokenizer.py and train_gpt.py via accelerate
  • For Meta-World MBRL, install the pinned Farama-Foundation/Metaworld commit and run mbrl/train_metaworld_mbpo.py

[2]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
iVideoGPT NeurIPS 2024 evaluations (arXiv:2405.15223v3)Upstream abstract reports competitive performance across action-conditioned video prediction (BAIR, RoboNet), visual planning (RoboSuite, RoboDesk), and visual model-based RL (Meta-World) after pre-training on millions of human and robotic manipulation trajectories.Upstream-reported paper abstract summary; numerical FVD/PSNR and RL return tables are in the paper body (not excerpted here) and not independently verified by OSR.

[2] [4]

Project notes

Checkpoints, resolution tradeoffs, and upstream caveats

Nine ungated, MIT-licensed safetensors checkpoints (1.01 GB to 2.20 GB each) are hosted on Hugging Face: four Open X-Embodiment pre-trained models (oxe-64-act-free, oxe-64-act-free-medium, oxe-64-goal-cond, oxe-256-act-free) and five downstream fine-tuned checkpoints covering BAIR, RoboNet, RoboSuite, and RoboDesk. Upstream documents three caveats: (1) there is no action-conditioned Open X-Embodiment pre-trained checkpoint due to heterogeneous action spaces across OXE datasets; (2) the 256x256 OXE checkpoint may not perform best due to insufficient training, though it can serve as a starting point for downstream fine-tuning; and (3) RoboNet 256x256 checkpoints were accidentally deleted during a disk cleanup. As an editorial starting recommendation, adopters should begin with inference/predict.py on the predominantly 64x64 releases for initial action-free exploration, while target-action-conditioned prediction requires either a matching downstream checkpoint (BAIR, RoboNet, RoboSuite, RoboDesk) or adaptation through the documented two-stage fine-tuning workflow (train_tokenizer.py and train_gpt.py). [2] [3]

Workflows and upstream-reported evaluation

The repository provides conversion scripts for Open X-Embodiment (OXE_SELECT) and benchmark datasets, accelerate training scripts (train_tokenizer.py and train_gpt.py in bf16), FVD evaluation (requiring an external I3D TorchScript weight), and visual control via mbrl/train_metaworld_mbpo.py (pinned to a specific Farama Meta-World commit) and vp/. Minimum GPU VRAM and inference latency are not quantified in the README. In arXiv:2405.15223v3, the authors report competitive results across action-conditioned video prediction, visual planning, and model-based RL; all evaluation claims are upstream-reported, and Open Source Robotics has not run training, fine-tuning, or inference on iVideoGPT. [2] [4]

What we checked

None. Prepared from public upstream sources without hands-on training, fine-tuning, or inference.

Updates

Upstream last updated
Sep 23, 2025 [1]
Latest release
Jun 3, 2025 [3]
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
thuml/iVideoGPT@d601d5cac9e96c6aa0c17cb37ed6a7c7ca1fb210; HF ivideogpt-oxe-64-act-free@dbca9c7c335f5924459b3ee9837f9e3fad11b1f3 · software repository

Separate license matrix

MIT-licensed NeurIPS 2024 robotics world model releasing pre-training, fine-tuning, evaluation, visual planning, and MBRL code alongside 9 ungated MIT safetensors checkpoints on Hugging Face; marked include_partial because pre-training and benchmark datasets are external third-party corpora with heterogeneous licenses not individually audited here.

LayerAvailability / license
Codeavailable · MIT
Verified MIT in root LICENSE; includes pre-training, fine-tuning, evaluation, visual planning, and MBRL code. [1]
Model weightsavailable · MIT
All 9 Hugging Face safetensors repositories are ungated and tagged license: mit. [3]
Training datasetsavailable · License not fully assessed
Pre-training and fine-tuning rely on external public datasets (OXE_SELECT in Open X-Embodiment, BAIR, RoboNet, RoboSuite, RoboDesk, Meta-World) under heterogeneous third-party licenses not individually audited here; conversion scripts are provided in datasets/. [2]
Documentationavailable · MIT
[1] [2]

Licensing and maintenance

Both the GitHub repository (thuml/iVideoGPT, HEAD d601d5cac9e9, 2025-09-23) and all nine Hugging Face checkpoints are licensed under the MIT License. External datasets (Open X-Embodiment, BAIR, RoboNet, RoboSuite, RoboDesk, Meta-World) retain their respective upstream terms; individual dataset licenses were not audited in this review and require separate verification for the chosen workflow. Subsequent research from the same lab is directed to thuml/RLVR-World. [1] [2] [3]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from GitHub commit d601d5cac9e9, NeurIPS 2024 arXiv:2405.15223v3, and 9 verified MIT Hugging Face checkpoints.