Video Prediction Policy (VPP)
Robotics modelTwo-stage generalist visuomotor policy pairing a manipulation-finetuned 1.52B Stable Video Diffusion backbone (predictive visual representations) with a Diffusion Transformer inverse-dynamics action model, with MIT-licensed code and ungated Hugging Face checkpoints. [1] [2] [3] [4] [5]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Robotics and embodied AI researchers; Video-conditioned policy and CALVIN practitioners; Dexterous manipulation researchers (RobotEra X-Bot / X-Hand) [1]
- Parameters
- 1,524,623,082 (~1.52B F32 params for SVD video backbone per HF safetensors metadata) + ~1.02 GB action policy… [1] [3] [5]
- Embodiments
- Franka Emika Panda (CALVIN ABC->D simulation), RobotEra X-Bot / X-Hand dexterous manipulation platform (training &… [1]
- Source rights
- Named licenses are documented by part. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Overview
Video Prediction Policy (VPP, ICML Spotlight / arXiv:2412.14803) is a two-stage visuomotor policy that fine-tunes a Stable Video Diffusion (SVD) model for text-guided manipulation video prediction and conditions a Diffusion Transformer inverse-dynamics action policy on its internal predictive visual representations. The official PyTorch implementation (roboterax/video-prediction-policy at commit 75cf37ad0f53) is MIT-licensed and pairs with three ungated Hugging Face checkpoints (yjguo/svd-robot, yjguo/svd-robot-calvin-ft, and yjguo/dp-calvin) plus pre-extracted SVD latent features (yjguo/vpp_svd_latent). [1] [2] [3] [4] [5] [6]
Model card summary
- Architecture
Two-stage architecture: (1) text-guided Stable Video Diffusion (SVD, 1.52B F32 parameters) finetuned for manipulation video prediction using a frozen CLIP ViT-B/32 text encoder; (2) Diffusion Transformer (MDT-based, DDIM 10 sampling steps, latent_dim 384) implicit inverse-dynamics action model conditioned on internal SVD predictive visual features.
- Base model
Stable Video Diffusion (Stability AI) + openai/clip-vit-base-patch32 + MDT diffusion transformer
- Parameter count
1,524,623,082 (~1.52B F32 params for SVD video backbone per HF safetensors metadata) + ~1.02 GB action policy checkpoint + frozen CLIP ViT-B/32 (~151M params; README lists ~600M download)
Exact SVD tensor count verified via HF API safetensors metadata on yjguo/svd-robot and yjguo/svd-robot-calvin-ft.
- Input modalities
- rgbproprioceptionlanguage
RGB camera observations, natural language task instructions (encoded via CLIP ViT-B/32), and robot proprioceptive state (used in X-Bot/CALVIN datasets).
- Output modalities
- ee-pose-deltagripper-commandvideo-frames
Stage 1 outputs predicted future RGB video frames/features; Stage 2 outputs 10-step continuous action chunks (act_seq_len: 10, DDIM 10 steps) for CALVIN 7-DoF manipulation or X-Bot/X-Hand control.
- Action space
Continuous diffusion action chunks (act_seq_len: 10, multistep: 10, DDIM sampler with 10 sampling steps in policy_conf/VPP_Calvinabc_train.yaml) for CALVIN end-effector/gripper control and RobotEra X-Bot/X-Hand dexterous manipulation.
- Trained / supported embodiments
- Franka Emika Panda (CALVIN ABC->D simulation)
- RobotEra X-Bot / X-Hand dexterous manipulation platform (training & deployment scripts included)
- Training datasets
- Something-Something-v2 (video pretraining)
- Open X-Embodiment subsets (Bridge, RT-1)
- CALVIN ABC-D
- RobotEra X-Hand demonstrations
- Pre-extracted latent features: yjguo/vpp_svd_latent (Hugging Face)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| yjguo/svd-robot (SVD video model finetuned on SthV2, OpenX, X-Hand) | safetensors | 9.02 GB | open | df40f0b30cf1 |
| yjguo/svd-robot-calvin-ft (SVD video model finetuned on SthV2, OpenX, CALVIN ABC) | safetensors | 9.02 GB | open | d7178b195d5f |
| yjguo/dp-calvin (CALVIN ABC action policy checkpoint) | pytorch | 1.02 GB | open | 199fc56a4e61 |
Exact recursive sizes verified via Hugging Face API (9.018 GB F32 safetensors each for svd-robot and svd-robot-calvin-ft; 1.020 GB last.pt for dp-calvin).
Is this for you?
- Use cases
- Reproducing CALVIN ABC->D manipulation rollouts with svd-robot-calvin-ft and dp-calvin
- Generating text-guided robot manipulation video predictions on Something-Something-v2, Bridge, RT-1, and X-Hand
- Fine-tuning VPP on custom robot embodiments using svd-robot and step2_train_action_xbot.py
- Known limitations
- Hugging Face model repositories (yjguo/svd-robot, yjguo/svd-robot-calvin-ft, yjguo/dp-calvin) do not declare an explicit license tag or LICENSE file, and the video backbone derives from Stable Video Diffusion.
- Training experiments require 8x A800/H100 GPUs and ~500 GB disk space for the CALVIN ABC-D dataset.
- Only the CALVIN action policy checkpoint (dp-calvin) is published; real-world RobotEra X-Bot/X-Hand action weights are not released.
- Safety and limitations
- Hugging Face model repos (yjguo/svd-robot, yjguo/svd-robot-calvin-ft, yjguo/dp-calvin) omit explicit license metadata, and the video backbone derives from Stable Video Diffusion whose upstream Stability AI Community License carries commercial revenue terms.
- Policy performance depends heavily on Stage-1 video prediction quality; the authors advise visually inspecting generated videos before training the action head.
- Real-world RobotEra X-Bot/X-Hand action weights are not released, limiting direct out-of-the-box evaluation to CALVIN and video prediction.
Compute & fine-tuning
- Inference compute
- Documented hardware: 8x NVIDIA A800/H100 node documented for training
- Precision:
fp32 - Edge / consumer capable: No
SVD checkpoints are stored in F32 safetensors (9.02 GB) alongside the 1.02 GB action model; exact minimum inference VRAM and latency_ms are not documented.
- Fine-tuning
- Supported: Yes
- Methods: Stage-1 SVD latent extraction and video fine-tuning (step1_train_svd.py); Stage-2 action diffusion transformer training on CALVIN or custom X-Bot/X-Hand datasets
- Recipe: Upstream fine-tuning documentation
- Host computer
Paper training runs used one node with 8x NVIDIA A800/H100 GPUs; inference requires loading both the ~9 GB F32 SVD video model and the ~1 GB action model plus CLIP text encoder.
- Operating system
- Linux
CUDA, Accelerate multi-GPU training, and CALVIN target Linux.
[1] · Editorial assessment
- Dependencies
- Python 3.10
- PyTorch (>=2.0 or 1.13 for CALVIN)
- Hugging Face Accelerate
- openai/clip-vit-base-patch32
- CALVIN simulator (for CALVIN evaluation)
- Skills
- Python 3.10
- PyTorch & Hugging Face Accelerate
- Video diffusion model fine-tuning
- CALVIN simulation setup
Inferred from documented conda, Accelerate, SVD, and CALVIN workflows.
[1] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [1]
- Create a Python 3.10 conda environment, optionally install CALVIN, and run pip install -r requirements.txt.
- Download openai/clip-vit-base-patch32, yjguo/svd-robot-calvin-ft, and yjguo/dp-calvin from Hugging Face.
- Run policy_evaluation/calvin_evaluate.py for CALVIN ABC->D rollouts or make_prediction.py for video predictions.
- For training, extract or download pre-extracted latents (yjguo/vpp_svd_latent) and launch step1_train_svd.py followed by step2_train_action_calvin.py or step2_train_action_xbot.py via Accelerate.
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| CALVIN ABC->D (zero-shot unseen environment) | Avg Len 4.33 | Upstream-reported simulation score from README and arXiv:2412.14803; not independently reproduced by OSR. |
| Real-world dexterous hand manipulation (RobotEra X-Hand) | 100+ tasks solved with a single VPP policy (per README) | Upstream-reported real-robot claim in README and arXiv:2412.14803; real-world action checkpoint is not published and results were not verified by OSR. |
Project notes
Two-stage SVD backbone and Diffusion Transformer action head
Stage 1 adapts a Stable Video Diffusion backbone (Hugging Face API safetensors metadata reports 1,524,623,082 F32 parameters—whose component coverage across submodules is unverified—while the combined safetensors files across image_encoder, unet, and vae total 9.02 GB) with a frozen openai/clip-vit-base-patch32 text encoder into two separate video checkpoints: yjguo/svd-robot (finetuned on Something-Something-v2, Open X-Embodiment subsets including Bridge and RT-1, and RobotEra X-Hand) for general video prediction and custom-robot training, and yjguo/svd-robot-calvin-ft (finetuned on Something-Something-v2, Open X-Embodiment, and CALVIN ABC video) for replicating CALVIN ABC rollouts. Stage 2 trains an MDT-based Diffusion Transformer action policy (policy_conf/VPP_Calvinabc_train.yaml: latent_dim: 384, act_seq_len: 10, multistep: 10, DDIM sampler with 10 sampling steps) that maps predictive visual features to continuous 10-step action chunks. Alongside CALVIN scripts, the repository provides dataset, config, training, and real-robot deployment code (step2_train_action_xbot.py, step3_deploy_real_xbot.py) for RobotEra's X-Bot/X-Hand platform, although only the CALVIN action checkpoint (yjguo/dp-calvin, 1.02 GB last.pt) is publicly hosted. [1] [3] [4] [5]
Compute footprint and upstream-reported benchmarks
The authors ran training on a single node with 8x NVIDIA A800/H100 GPUs and note that full CALVIN ABC-D reproduction requires ~500 GB of dataset storage; exact minimum inference VRAM and closed-loop control frequency are not stated in the README. The repository README reports an average sequence length of 4.33 on the simulated CALVIN ABC->D benchmark and claims 100+ real-world dexterous hand manipulation tasks solved with a single VPP policy (while arXiv:2412.14803 summarizes broader simulation and real-world improvements). These metrics are strictly upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [6]
What we checked
None. Prepared from public upstream sources without hands-on training, fine-tuning, or simulation execution.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Upstream last updated
- May 17, 2025 [1]
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- roboterax/video-prediction-policy@75cf37ad0f537627e1a3aa9a04bab70324ec27ff; HF yjguo/svd-robot@df40f0b30cf1ff486e18550a61f12ffc52838e38; HF yjguo/svd-robot-calvin-ft@d7178b195d5f934f5d5b268e25fdf011f16b7bfb; HF yjguo/dp-calvin@199fc56a4e61fd72df55d1fe5dacd293504b42e4 · combined hardware and software repository
Separate license matrix
Publishes MIT-licensed training/evaluation code and three ungated Hugging Face checkpoints (1.52B SVD video backbones and CALVIN action policy) plus pre-extracted latents, but the Hugging Face weight repositories omit explicit license declarations and derive from Stable Video Diffusion.
| Layer | Availability / license |
|---|---|
| Code | available · MIT Verified MIT License (Copyright 2025 Tsinghua University) in root LICENSE file. [1] [2] |
| Model weights | available · Unspecified on Hugging Face Hub (derived from Stable Video Diffusion; upstream SVD license terms apply) All three Hugging Face repos (svd-robot, svd-robot-calvin-ft, dp-calvin) are ungated and downloadable, but none declares a license field or LICENSE file. [3] [4] [5] |
| Training datasets | available · Mixed (upstream SthV2, Open X-Embodiment, CALVIN licenses; X-Hand data not publicly released) Pre-extracted SVD latent features for SthV2, Bridge, RT-1, and CALVIN are hosted on Hugging Face. [1] |
| Documentation | available · MIT [1] [2] |
Licensing gap between code and weights
While roboterax/video-prediction-policy carries a clear MIT License (Copyright 2025 Tsinghua University), none of the three Hugging Face checkpoint repositories (yjguo/svd-robot, yjguo/svd-robot-calvin-ft, yjguo/dp-calvin) declares a license field or includes a LICENSE file. Because the video models are fine-tuned from Stability AI's Stable Video Diffusion and pretraining spans heterogeneous datasets (Something-Something-v2, Open X-Embodiment, CALVIN, and unreleased X-Hand demonstrations), downstream users must verify upstream SVD and dataset terms before commercial use. [1] [2] [3] [4] [5]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 roboterax/video-prediction-policy GitHub repository README.md and policy_conf/VPP_Calvinabc_train.yaml
Documents two-stage SVD + MDT diffusion transformer architecture, act_seq_len 10 / DDIM 10 steps, CALVIN ABC Avg Len 4.33, 8x A800/H100 training, and Hugging Face checkpoint/latent links.Revision, date & where cited
roboterax (GitHub) · github.com · revision
75cf37a· README.md and policy_conf/VPP_Calvinabc_train.yaml at commit 75cf37ad0f53 (2025-05-17) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Compute & fine-tuning and 6 more - S002 roboterax/video-prediction-policy root LICENSE file
MIT License, Copyright (c) 2025 Tsinghua University.Revision, date & where cited
roboterax (GitHub) · github.com · revision
75cf37a· LICENSE at commit 75cf37ad0f53 · Accessed Nov 21, 2026
Used for: Overview, Separate license matrix - S003 Hugging Face API metadata: yjguo/svd-robot
1,524,623,082 F32 parameters in safetensors (9.018 GB total across image_encoder, unet, vae), ungated, no license declared.Revision, date & where cited
Hugging Face (yjguo) · huggingface.co · HF API at revision df40f0b30cf1 (2025-05-04) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 3 more - S004 Hugging Face API metadata: yjguo/svd-robot-calvin-ft
1,524,623,082 F32 parameters in safetensors (9.018 GB total), ungated, no license declared.Revision, date & where cited
Hugging Face (yjguo) · huggingface.co · HF API at revision d7178b195d5f (2025-05-06) · Accessed Nov 21, 2026
Used for: Overview, Checkpoints, Is this for you?, Two-stage SVD backbone and Diffusion Transformer action head and 2 more - S005 Hugging Face API metadata: yjguo/dp-calvin
Ungated CALVIN action policy checkpoint (last.pt, 1.020 GB), no license declared.Revision, date & where cited
Hugging Face (yjguo) · huggingface.co · HF API at revision 199fc56a4e61 (2025-05-06) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 4 more - S006 arXiv:2412.14803 — Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations
ICML Spotlight paper introducing VPP using video diffusion model predictive visual representations for robotic control.Revision, date & where cited
arXiv · arxiv.org · arXiv:2412.14803v2 (2024-12-19) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Upstream-reported benchmarks, Compute footprint and upstream-reported benchmarks
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from roboterax/video-prediction-policy@75cf37ad0f53 and yjguo HF checkpoints.