iVideoGPT
Robotics modelMIT-licensed autoregressive transformer world model (NeurIPS 2024, Tsinghua THUML) combining a compressive context-aware VQ-GAN tokenizer (114M/310M) and Llama-style GPT (138M/436M), pre-trained on Open X-Embodiment with 9 ungated Hugging Face checkpoints. [1] [2] [3] [4]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Model-based reinforcement learning researchers; Robotics world-model and video-prediction researchers; Open-weights robotics practitioners seeking lightweight MIT checkpoints [2] [3] [4]
- Parameters
- 252M to 550M total (114M tokenizer + 138M transformer = 252M [2]
- Embodiments
- Open X-Embodiment multi-robot manipulation datasets (action-free and goal-conditioned pre-training), BAIR robot… [2]
- Source rights
- Some licenses are unresolved. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- License
- MIT · license not fully assessed for datasets · Details by part
Overview
iVideoGPT (NeurIPS 2024, Tsinghua THUML) is a scalable autoregressive Transformer world model that interleaves visual, action, and reward tokens for next-token prediction. Its architecture pairs a compressive context-aware VQ-GAN tokenizer (ctx_vqgan, 114M parameters at 64x64 or 310M at 256x256) with a Llama-architecture GPT backbone (138M or 436M parameters), yielding 252M–550M total parameters across the released variants. The thuml/iVideoGPT repository provides pre-training, two-stage fine-tuning, evaluation, visual planning, and Meta-World model-based RL (MBPO) code under the MIT License. [1] [2] [4]
Model card summary
- Architecture
Autoregressive Transformer (Llama-style GPT, 138M or 436M) paired with a compressive context-aware VQ-GAN video tokenizer (ctx_vqgan, 114M at 64x64 or 310M at 256x256) that interleaves visual, action, and reward tokens for next-token world-model prediction.
- Base model
Trained from scratch (ctx_vqgan tokenizer + Llama-architecture GPT transformer)
- Parameter count
252M to 550M total (114M tokenizer + 138M transformer = 252M; 114M + 436M = 550M medium; 310M + 138M = 448M at 256x256)
- Input modalities
- rgbproprioception
Visual RGB frames (64x64 or 256x256), plus optional low-dimensional robot actions, rewards, or goal images in conditioned variants.
- Output modalities
- video-frames
Generates future RGB video frames (64x64 or 256x256) and reward tokens in MBRL mode.
- Action space
Action-free or goal-conditioned on Open X-Embodiment; environment-specific low-dimensional continuous actions (e.g., --action_dim 4 for BAIR) in fine-tuned action-conditioned checkpoints, visual planning, and Meta-World MBRL.
- Control frequency
Not specified (operates on discrete dataset step horizons, e.g., segment_length 8-16 and video_stepsize 1)
- Trained / supported embodiments
- Open X-Embodiment multi-robot manipulation datasets (action-free and goal-conditioned pre-training)
- BAIR robot pushing and RoboNet multi-robot arms
- RoboSuite, RoboDesk, and Meta-World simulated manipulation benchmarks
- Training datasets
- Open X-Embodiment (OXE_SELECT subset) for pre-training
- BAIR Robot Pushing, RoboNet, RoboSuite, RoboDesk, and Meta-World for downstream fine-tuning and control
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| thuml/ivideogpt-oxe-64-act-free | safetensors | 1.01 GB | open | dbca9c7c335f |
| thuml/ivideogpt-oxe-64-act-free-medium | safetensors | 2.2 GB | open | 0076304ff769 |
| thuml/ivideogpt-oxe-64-goal-cond | safetensors | 1.01 GB | open | 4d8dfae1e4cb |
| thuml/ivideogpt-oxe-256-act-free | safetensors | 1.8 GB | open | 675bfd148276 |
| thuml/ivideogpt-bair-64-act-free | safetensors | 1.01 GB | open | 93e00a1f9981 |
| thuml/ivideogpt-bair-64-act-cond | safetensors | 1.01 GB | open | 2863382adad1 |
| thuml/ivideogpt-robonet-64-act-cond | safetensors | 1.01 GB | open | eb88e1481f49 |
| thuml/ivideogpt-vp2-robosuite-64-act-cond | safetensors | 1.01 GB | open | dfe4c1db8777 |
| thuml/ivideogpt-vp2-robodesk-64-act-cond | safetensors | 1.01 GB | open | 6f974cfbee8f |
Sizes are decimal GB sums of sibling safetensors/config files at each pinned HF revision. All 9 checkpoints are ungated and MIT-licensed.
Is this for you?
- Use cases
- Action-free and goal-conditioned video prediction pre-trained on Open X-Embodiment
- Fine-tuned action-conditioned video prediction (BAIR, RoboNet, RoboSuite, RoboDesk)
- Visual model-based RL (MBPO on Meta-World) and visual planning
- Known limitations
- No action-conditioned Open X-Embodiment pre-trained checkpoint is provided due to heterogeneous action spaces across OXE datasets
- Released 256x256 OXE checkpoint is explicitly noted by upstream as under-trained (intended mainly as a fine-tuning starting point); most released checkpoints operate at 64x64 resolution
- RoboNet 256x256 checkpoints were accidentally deleted during a disk cleanup and are not available
- Minimum GPU VRAM requirements for pre-training, fine-tuning, and inference are not explicitly stated in the README
- Safety and limitations
- Outputs predicted low-resolution video frames (primarily 64x64) and imagined rollouts for planning/MBRL rather than safety-certified physical robot control.
- No action-conditioned OXE pre-trained model is available; users must fine-tune on their target action space.
- 256x256 OXE checkpoint is explicitly flagged upstream as insufficiently trained for best standalone performance, and RoboNet 256x256 checkpoints were lost during cleanup.
- Minimum VRAM and inference latency figures are not reported in the repository README.
Compute & fine-tuning
- Inference compute
- Documented hardware: Not specified in README (bf16 accelerate training documented; 1.01-2.20 GB checkpoint footprints)
- Precision:
bf16 - Edge / consumer capable: No
Exact minimum GPU VRAM and inference latency are not stated in the README.
- Fine-tuning
- Supported: Yes
- Methods: Two-stage fine-tuning (ctx_vqgan tokenizer via train_tokenizer.py + Llama transformer via train_gpt.py); Visual model-based RL (MBPO on Meta-World)
- Recipe: Upstream fine-tuning documentation
- Host computer
GPU-accelerated Python 3.9 environment (exact minimum VRAM not stated upstream; bf16 mixed precision used in documented accelerate fine-tuning commands; checkpoints are 1.01-2.20 GB safetensors).
- Operating system
- Linux
Inferred from shell scripts, Conda/CUDA workflow, and Linux MBRL paths.
[2] · Editorial assessment
- Dependencies
- Python 3.9
- PyTorch and Hugging Face Accelerate
- Hugging Face Diffusers (codebase base)
- DrQ-v2 / Meta-World (pinned commit 83ac03ca for MBRL)
- Pretrained I3D TorchScript model (optional for FVD evaluation)
- Skills
- Python 3.9 and Conda environment management
- PyTorch, Hugging Face Diffusers, and Accelerate
- RL / Meta-World setup for MBRL experiments
Inferred from README installation, accelerate launch commands, and MBRL setup.
[2] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [2] · Model card / documentation [1] [2]
- Create a Python 3.9 Conda environment and install dependencies from requirements.txt
- Run inference/predict.py pointing to any ungated Hugging Face checkpoint (e.g., thuml/ivideogpt-oxe-64-act-free)
- For training or fine-tuning, convert Open X-Embodiment or benchmark datasets with datasets/oxe_data_converter.py and launch train_tokenizer.py and train_gpt.py via accelerate
- For Meta-World MBRL, install the pinned Farama-Foundation/Metaworld commit and run mbrl/train_metaworld_mbpo.py
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| iVideoGPT NeurIPS 2024 evaluations (arXiv:2405.15223v3) | Upstream abstract reports competitive performance across action-conditioned video prediction (BAIR, RoboNet), visual planning (RoboSuite, RoboDesk), and visual model-based RL (Meta-World) after pre-training on millions of human and robotic manipulation trajectories. | Upstream-reported paper abstract summary; numerical FVD/PSNR and RL return tables are in the paper body (not excerpted here) and not independently verified by OSR. |
Project notes
Checkpoints, resolution tradeoffs, and upstream caveats
Nine ungated, MIT-licensed safetensors checkpoints (1.01 GB to 2.20 GB each) are hosted on Hugging Face: four Open X-Embodiment pre-trained models (oxe-64-act-free, oxe-64-act-free-medium, oxe-64-goal-cond, oxe-256-act-free) and five downstream fine-tuned checkpoints covering BAIR, RoboNet, RoboSuite, and RoboDesk. Upstream documents three caveats: (1) there is no action-conditioned Open X-Embodiment pre-trained checkpoint due to heterogeneous action spaces across OXE datasets; (2) the 256x256 OXE checkpoint may not perform best due to insufficient training, though it can serve as a starting point for downstream fine-tuning; and (3) RoboNet 256x256 checkpoints were accidentally deleted during a disk cleanup. As an editorial starting recommendation, adopters should begin with inference/predict.py on the predominantly 64x64 releases for initial action-free exploration, while target-action-conditioned prediction requires either a matching downstream checkpoint (BAIR, RoboNet, RoboSuite, RoboDesk) or adaptation through the documented two-stage fine-tuning workflow (train_tokenizer.py and train_gpt.py). [2] [3]
Workflows and upstream-reported evaluation
The repository provides conversion scripts for Open X-Embodiment (OXE_SELECT) and benchmark datasets, accelerate training scripts (train_tokenizer.py and train_gpt.py in bf16), FVD evaluation (requiring an external I3D TorchScript weight), and visual control via mbrl/train_metaworld_mbpo.py (pinned to a specific Farama Meta-World commit) and vp/. Minimum GPU VRAM and inference latency are not quantified in the README. In arXiv:2405.15223v3, the authors report competitive results across action-conditioned video prediction, visual planning, and model-based RL; all evaluation claims are upstream-reported, and Open Source Robotics has not run training, fine-tuning, or inference on iVideoGPT. [2] [4]
What we checked
None. Prepared from public upstream sources without hands-on training, fine-tuning, or inference.
Updates
- Upstream last updated
- Sep 23, 2025 [1]
- Latest release
- Jun 3, 2025 [3]
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- thuml/iVideoGPT@d601d5cac9e96c6aa0c17cb37ed6a7c7ca1fb210; HF ivideogpt-oxe-64-act-free@dbca9c7c335f5924459b3ee9837f9e3fad11b1f3 · software repository
Separate license matrix
MIT-licensed NeurIPS 2024 robotics world model releasing pre-training, fine-tuning, evaluation, visual planning, and MBRL code alongside 9 ungated MIT safetensors checkpoints on Hugging Face; marked include_partial because pre-training and benchmark datasets are external third-party corpora with heterogeneous licenses not individually audited here.
| Layer | Availability / license |
|---|---|
| Code | available · MIT Verified MIT in root LICENSE; includes pre-training, fine-tuning, evaluation, visual planning, and MBRL code. [1] |
| Model weights | available · MIT All 9 Hugging Face safetensors repositories are ungated and tagged license: mit. [3] |
| Training datasets | available · License not fully assessed Pre-training and fine-tuning rely on external public datasets (OXE_SELECT in Open X-Embodiment, BAIR, RoboNet, RoboSuite, RoboDesk, Meta-World) under heterogeneous third-party licenses not individually audited here; conversion scripts are provided in datasets/. [2] |
| Documentation | available · MIT [1] [2] |
Licensing and maintenance
Both the GitHub repository (thuml/iVideoGPT, HEAD d601d5cac9e9, 2025-09-23) and all nine Hugging Face checkpoints are licensed under the MIT License. External datasets (Open X-Embodiment, BAIR, RoboNet, RoboSuite, RoboDesk, Meta-World) retain their respective upstream terms; individual dataset licenses were not audited in this review and require separate verification for the chosen workflow. Subsequent research from the same lab is directed to thuml/RLVR-World. [1] [2] [3]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 thuml/iVideoGPT GitHub metadata, HEAD commit, and LICENSE
Default branch main HEAD is d601d5cac9e96c6aa0c17cb37ed6a7c7ca1fb210 (2025-09-23T13:45:51Z); archived=false; root LICENSE is MIT.Revision, date & where cited
thuml (GitHub) · github.com · revision
d601d5c· GitHub API repos/thuml/iVideoGPT & root LICENSE at d601d5cac9e9 · Accessed Nov 21, 2026
Used for: Overview, Setup & usage, What we checked, Separate license matrix - S002 thuml/iVideoGPT README.md at pinned commit
Lists 9 Hugging Face checkpoints (114M/310M tokenizer + 138M/436M transformer), OXE/BAIR/RoboNet/RoboSuite/RoboDesk/Meta-World workflows, action-free OXE caveat, under-trained 256x256 caveat, and lost RoboNet-256 checkpoints.Revision, date & where cited
thuml (GitHub) · github.com · revision
d601d5c· README.md at commit d601d5cac9e9 · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 6 more - S003 Hugging Face API metadata for 9 thuml/ivideogpt-* checkpoints
Verified pinned SHAs, sibling safetensors sizes (1.01 GB to 2.20 GB), gated=false, and license='mit' across all 9 checkpoints.Revision, date & where cited
thuml (Hugging Face) · huggingface.co · HF API /api/models/thuml/ivideogpt-* (9 repos) · Accessed Nov 21, 2026
Used for: Model card summary, Checkpoints, Compute & fine-tuning, Checkpoints, resolution tradeoffs, and upstream caveats and 2 more - S004 iVideoGPT: Interactive VideoGPTs are Scalable World Models (arXiv:2405.15223v3, NeurIPS 2024)
Describes autoregressive next-token world-model architecture with compressive tokenization pre-trained on millions of human and robotic manipulation trajectories for video prediction, visual planning, and MBRL.Revision, date & where cited
arXiv (NeurIPS 2024) · arxiv.org · arXiv abstract 2405.15223v3 (2024-10-31) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Upstream-reported benchmarks and 1 more
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from GitHub commit d601d5cac9e9, NeurIPS 2024 arXiv:2405.15223v3, and 9 verified MIT Hugging Face checkpoints.