Open Source RoboticsRobots, software & learning

Video Prediction Policy (VPP)

Robotics model

Two-stage generalist visuomotor policy pairing a manipulation-finetuned 1.52B Stable Video Diffusion backbone (predictive visual representations) with a Diffusion Transformer inverse-dynamics action model, with MIT-licensed code and ungated Hugging Face checkpoints. [1] [2] [3] [4] [5]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Robotics and embodied AI researchers; Video-conditioned policy and CALVIN practitioners; Dexterous manipulation researchers (RobotEra X-Bot / X-Hand) [1]
Parameters
1,524,623,082 (~1.52B F32 params for SVD video backbone per HF safetensors metadata) + ~1.02 GB action policy… [1] [3] [5]
Embodiments
Franka Emika Panda (CALVIN ABC->D simulation), RobotEra X-Bot / X-Hand dexterous manipulation platform (training &… [1]
Source rights
Named licenses are documented by part. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware

Requirements · Research details and dates

On this page

Overview

Video Prediction Policy (VPP, ICML Spotlight / arXiv:2412.14803) is a two-stage visuomotor policy that fine-tunes a Stable Video Diffusion (SVD) model for text-guided manipulation video prediction and conditions a Diffusion Transformer inverse-dynamics action policy on its internal predictive visual representations. The official PyTorch implementation (roboterax/video-prediction-policy at commit 75cf37ad0f53) is MIT-licensed and pairs with three ungated Hugging Face checkpoints (yjguo/svd-robot, yjguo/svd-robot-calvin-ft, and yjguo/dp-calvin) plus pre-extracted SVD latent features (yjguo/vpp_svd_latent). [1] [2] [3] [4] [5] [6]

Model card summary

Architecture

Two-stage architecture: (1) text-guided Stable Video Diffusion (SVD, 1.52B F32 parameters) finetuned for manipulation video prediction using a frozen CLIP ViT-B/32 text encoder; (2) Diffusion Transformer (MDT-based, DDIM 10 sampling steps, latent_dim 384) implicit inverse-dynamics action model conditioned on internal SVD predictive visual features.

[1] [3] [6]

Base model

Stable Video Diffusion (Stability AI) + openai/clip-vit-base-patch32 + MDT diffusion transformer

[1]

Parameter count

1,524,623,082 (~1.52B F32 params for SVD video backbone per HF safetensors metadata) + ~1.02 GB action policy checkpoint + frozen CLIP ViT-B/32 (~151M params; README lists ~600M download)

Exact SVD tensor count verified via HF API safetensors metadata on yjguo/svd-robot and yjguo/svd-robot-calvin-ft.

[1] [3] [5]

Input modalities
rgbproprioceptionlanguage

RGB camera observations, natural language task instructions (encoded via CLIP ViT-B/32), and robot proprioceptive state (used in X-Bot/CALVIN datasets).

[1]

Output modalities
ee-pose-deltagripper-commandvideo-frames

Stage 1 outputs predicted future RGB video frames/features; Stage 2 outputs 10-step continuous action chunks (act_seq_len: 10, DDIM 10 steps) for CALVIN 7-DoF manipulation or X-Bot/X-Hand control.

[1]

Action space

Continuous diffusion action chunks (act_seq_len: 10, multistep: 10, DDIM sampler with 10 sampling steps in policy_conf/VPP_Calvinabc_train.yaml) for CALVIN end-effector/gripper control and RobotEra X-Bot/X-Hand dexterous manipulation.

[1]

Trained / supported embodiments
  • Franka Emika Panda (CALVIN ABC->D simulation)
  • RobotEra X-Bot / X-Hand dexterous manipulation platform (training & deployment scripts included)

[1]

Training datasets
  • Something-Something-v2 (video pretraining)
  • Open X-Embodiment subsets (Bridge, RT-1)
  • CALVIN ABC-D
  • RobotEra X-Hand demonstrations
  • Pre-extracted latent features: yjguo/vpp_svd_latent (Hugging Face)

[1]

Interfaces
  • PyTorch / Hydra CLI (step1_train_svd.py, step2_train_action_calvin.py, step2_train_action_xbot.py)
  • Real-robot deployment agent (step3_deploy_real_xbot.py)
  • Hugging Face Diffusers safetensors (SVD) + PyTorch .pt checkpoint (action policy)

[1] [3] [5]

Checkpoints

CheckpointFormatSizeAccessPinned revision
yjguo/svd-robot (SVD video model finetuned on SthV2, OpenX, X-Hand)safetensors9.02 GBopendf40f0b30cf1
yjguo/svd-robot-calvin-ft (SVD video model finetuned on SthV2, OpenX, CALVIN ABC)safetensors9.02 GBopend7178b195d5f
yjguo/dp-calvin (CALVIN ABC action policy checkpoint)pytorch1.02 GBopen199fc56a4e61

Exact recursive sizes verified via Hugging Face API (9.018 GB F32 safetensors each for svd-robot and svd-robot-calvin-ft; 1.020 GB last.pt for dp-calvin).

[3] [4] [5]

Is this for you?

Use cases
  • Reproducing CALVIN ABC->D manipulation rollouts with svd-robot-calvin-ft and dp-calvin
  • Generating text-guided robot manipulation video predictions on Something-Something-v2, Bridge, RT-1, and X-Hand
  • Fine-tuning VPP on custom robot embodiments using svd-robot and step2_train_action_xbot.py

[1]

Known limitations
  • Hugging Face model repositories (yjguo/svd-robot, yjguo/svd-robot-calvin-ft, yjguo/dp-calvin) do not declare an explicit license tag or LICENSE file, and the video backbone derives from Stable Video Diffusion.
  • Training experiments require 8x A800/H100 GPUs and ~500 GB disk space for the CALVIN ABC-D dataset.
  • Only the CALVIN action policy checkpoint (dp-calvin) is published; real-world RobotEra X-Bot/X-Hand action weights are not released.

[1] [3] [4] [5]

Safety and limitations
  • Hugging Face model repos (yjguo/svd-robot, yjguo/svd-robot-calvin-ft, yjguo/dp-calvin) omit explicit license metadata, and the video backbone derives from Stable Video Diffusion whose upstream Stability AI Community License carries commercial revenue terms.
  • Policy performance depends heavily on Stage-1 video prediction quality; the authors advise visually inspecting generated videos before training the action head.
  • Real-world RobotEra X-Bot/X-Hand action weights are not released, limiting direct out-of-the-box evaluation to CALVIN and video prediction.

[1] [3] [4] [5]

Compute & fine-tuning

Inference compute
  • Documented hardware: 8x NVIDIA A800/H100 node documented for training
  • Precision: fp32
  • Edge / consumer capable: No

SVD checkpoints are stored in F32 safetensors (9.02 GB) alongside the 1.02 GB action model; exact minimum inference VRAM and latency_ms are not documented.

[1] [3] [5] · Editorial assessment

Fine-tuning
  • Supported: Yes
  • Methods: Stage-1 SVD latent extraction and video fine-tuning (step1_train_svd.py); Stage-2 action diffusion transformer training on CALVIN or custom X-Bot/X-Hand datasets
  • Recipe: Upstream fine-tuning documentation

[1]

Host computer

Paper training runs used one node with 8x NVIDIA A800/H100 GPUs; inference requires loading both the ~9 GB F32 SVD video model and the ~1 GB action model plus CLIP text encoder.

[1] [3] [5]

Operating system
  • Linux

CUDA, Accelerate multi-GPU training, and CALVIN target Linux.

[1] · Editorial assessment

Dependencies
  • Python 3.10
  • PyTorch (>=2.0 or 1.13 for CALVIN)
  • Hugging Face Accelerate
  • openai/clip-vit-base-patch32
  • CALVIN simulator (for CALVIN evaluation)

[1]

Skills
  • Python 3.10
  • PyTorch & Hugging Face Accelerate
  • Video diffusion model fine-tuning
  • CALVIN simulation setup

Inferred from documented conda, Accelerate, SVD, and CALVIN workflows.

[1] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1] · Model card / documentation [1]

  • Create a Python 3.10 conda environment, optionally install CALVIN, and run pip install -r requirements.txt.
  • Download openai/clip-vit-base-patch32, yjguo/svd-robot-calvin-ft, and yjguo/dp-calvin from Hugging Face.
  • Run policy_evaluation/calvin_evaluate.py for CALVIN ABC->D rollouts or make_prediction.py for video predictions.
  • For training, extract or download pre-extracted latents (yjguo/vpp_svd_latent) and launch step1_train_svd.py followed by step2_train_action_calvin.py or step2_train_action_xbot.py via Accelerate.

[1]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
CALVIN ABC->D (zero-shot unseen environment)Avg Len 4.33Upstream-reported simulation score from README and arXiv:2412.14803; not independently reproduced by OSR.
Real-world dexterous hand manipulation (RobotEra X-Hand)100+ tasks solved with a single VPP policy (per README)Upstream-reported real-robot claim in README and arXiv:2412.14803; real-world action checkpoint is not published and results were not verified by OSR.

[1] [6]

Project notes

Two-stage SVD backbone and Diffusion Transformer action head

Stage 1 adapts a Stable Video Diffusion backbone (Hugging Face API safetensors metadata reports 1,524,623,082 F32 parameters—whose component coverage across submodules is unverified—while the combined safetensors files across image_encoder, unet, and vae total 9.02 GB) with a frozen openai/clip-vit-base-patch32 text encoder into two separate video checkpoints: yjguo/svd-robot (finetuned on Something-Something-v2, Open X-Embodiment subsets including Bridge and RT-1, and RobotEra X-Hand) for general video prediction and custom-robot training, and yjguo/svd-robot-calvin-ft (finetuned on Something-Something-v2, Open X-Embodiment, and CALVIN ABC video) for replicating CALVIN ABC rollouts. Stage 2 trains an MDT-based Diffusion Transformer action policy (policy_conf/VPP_Calvinabc_train.yaml: latent_dim: 384, act_seq_len: 10, multistep: 10, DDIM sampler with 10 sampling steps) that maps predictive visual features to continuous 10-step action chunks. Alongside CALVIN scripts, the repository provides dataset, config, training, and real-robot deployment code (step2_train_action_xbot.py, step3_deploy_real_xbot.py) for RobotEra's X-Bot/X-Hand platform, although only the CALVIN action checkpoint (yjguo/dp-calvin, 1.02 GB last.pt) is publicly hosted. [1] [3] [4] [5]

Compute footprint and upstream-reported benchmarks

The authors ran training on a single node with 8x NVIDIA A800/H100 GPUs and note that full CALVIN ABC-D reproduction requires ~500 GB of dataset storage; exact minimum inference VRAM and closed-loop control frequency are not stated in the README. The repository README reports an average sequence length of 4.33 on the simulated CALVIN ABC->D benchmark and claims 100+ real-world dexterous hand manipulation tasks solved with a single VPP policy (while arXiv:2412.14803 summarizes broader simulation and real-world improvements). These metrics are strictly upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [6]

What we checked

None. Prepared from public upstream sources without hands-on training, fine-tuning, or simulation execution.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
May 17, 2025 [1]
Latest release
May 6, 2025 [4] [5]
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
roboterax/video-prediction-policy@75cf37ad0f537627e1a3aa9a04bab70324ec27ff; HF yjguo/svd-robot@df40f0b30cf1ff486e18550a61f12ffc52838e38; HF yjguo/svd-robot-calvin-ft@d7178b195d5f934f5d5b268e25fdf011f16b7bfb; HF yjguo/dp-calvin@199fc56a4e61fd72df55d1fe5dacd293504b42e4 · combined hardware and software repository

Separate license matrix

Publishes MIT-licensed training/evaluation code and three ungated Hugging Face checkpoints (1.52B SVD video backbones and CALVIN action policy) plus pre-extracted latents, but the Hugging Face weight repositories omit explicit license declarations and derive from Stable Video Diffusion.

LayerAvailability / license
Codeavailable · MIT
Verified MIT License (Copyright 2025 Tsinghua University) in root LICENSE file. [1] [2]
Model weightsavailable · Unspecified on Hugging Face Hub (derived from Stable Video Diffusion; upstream SVD license terms apply)
All three Hugging Face repos (svd-robot, svd-robot-calvin-ft, dp-calvin) are ungated and downloadable, but none declares a license field or LICENSE file. [3] [4] [5]
Training datasetsavailable · Mixed (upstream SthV2, Open X-Embodiment, CALVIN licenses; X-Hand data not publicly released)
Pre-extracted SVD latent features for SthV2, Bridge, RT-1, and CALVIN are hosted on Hugging Face. [1]
Documentationavailable · MIT
[1] [2]

Licensing gap between code and weights

While roboterax/video-prediction-policy carries a clear MIT License (Copyright 2025 Tsinghua University), none of the three Hugging Face checkpoint repositories (yjguo/svd-robot, yjguo/svd-robot-calvin-ft, yjguo/dp-calvin) declares a license field or includes a LICENSE file. Because the video models are fine-tuned from Stability AI's Stable Video Diffusion and pretraining spans heterogeneous datasets (Something-Something-v2, Open X-Embodiment, CALVIN, and unreleased X-Hand demonstrations), downstream users must verify upstream SVD and dataset terms before commercial use. [1] [2] [3] [4] [5]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from roboterax/video-prediction-policy@75cf37ad0f53 and yjguo HF checkpoints.