Open Source RoboticsRobots, software & learning

V-JEPA 2-AC

Robotics model

Action-conditioned video world-model implementation and DROID post-training configuration for robotics research. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Robotics researchers evaluating action-conditioned latent prediction [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]
Parameters
1B ViT-g/16 encoder (1,034,555,264 F32 params in base HF checkpoint) + 24-layer d=1024 action-conditioned predictor… [1] [6] [12] [14]
Embodiments
Franka Emika Panda (DROID post-training dataset and zero-shot two-lab evaluations reported in arXiv:2506.09985) [6] [13]
Source rights
Some licenses are unresolved. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware
License
MIT majority code; selected preprocessing Apache-2.0 · license not fully assessed for model weights · Details by part

Requirements · Research details and dates

On this page

Overview

V-JEPA 2-AC is Meta FAIR's action-conditioned latent world model inside the facebookresearch/vjepa2 repository (which also hosts generic V-JEPA 2 and V-JEPA 2.1 video encoders). Rather than decoding robot actions directly, V-JEPA 2-AC pairs a frozen 1B-parameter V-JEPA 2 ViT-g/16 video encoder (1,034,555,264 F32 parameters in the base facebook/vjepa2-vitg-fpc64-256 Hugging Face checkpoint) with a 24-layer action-conditioned transformer predictor (pred_embed_dim: 1024, pred_num_heads: 16, RoPE) to predict future latent video representations given robot actions. [1] [5] [6] [13] [14]

Model card summary

Architecture

Frozen V-JEPA 2 ViT-g/16 video encoder (1B / 1,034,555,264 params in base HF ViT-g) paired with a 24-layer action-conditioned transformer predictor (pred_embed_dim: 1024, pred_num_heads: 16, RoPE) operating on 256x256 8-frame video clips.

[1] [5] [6] [14]

Base model

V-JEPA 2 ViT-g/16 (facebook/vjepa2-vitg-fpc64-256, 1B parameters)

[1] [10] [14]

Parameter count

1B ViT-g/16 encoder (1,034,555,264 F32 params in base HF checkpoint) + 24-layer d=1024 action-conditioned predictor (exact combined AC parameter count not stated in README; vjepa2-ac-vitg.pt is 11.76 GB)

[1] [6] [12] [14]

Input modalities
rgbproprioception

256x256 RGB video clips (8 frames/clip in droid-256px-8f.yaml) conditioned on recorded robot end-effector state and action sequences.

[5] [6] [13]

Output modalities
video-frames

Predicts future latent video patch representations conditioned on robot actions (mapped to closest schema enum video-frames; outputs latent embeddings rather than decoded RGB pixels, used with planning for robot control).

[5] [6] [13] · Editorial assessment

Action space

Action-conditioned latent dynamics prediction over 256x256 8-frame clips conditioned on DROID robot trajectory actions; downstream manipulation relies on external image-goal planning rather than direct end-to-end policy action decoding.

[5] [6] [13]

Trained / supported embodiments
  • Franka Emika Panda (DROID post-training dataset and zero-shot two-lab evaluations reported in arXiv:2506.09985)

[6] [13]

Training datasets
  • V-JEPA 2 base encoder pretrained on >1M hours of internet video; V-JEPA 2-AC post-trained on <62 hours of unlabeled robot videos from the DROID dataset (per arXiv:2506.09985)

[6] [13]

Interfaces
  • PyTorch Hub (vjepa2_ac_vit_giant)
  • PyTorch .pt checkpoint
  • YAML training configs (app/vjepa_droid)

[1] [4] [7]

Checkpoints

CheckpointFormatSizeAccessPinned revision
vjepa2-ac-vitg.ptpytorch11.76 GBopenetag:12c945a28cc4f447e728fd2ba810607c-1402 (2025-05-30)

Public HTTP 200 download (11,760,743,310 bytes) verified via HEAD on 2026-11-21 without downloading binary payload.

[1] [11] [12]

Is this for you?

Use cases
  • Latent action-conditioned video dynamics prediction for robotics planning
  • Zero-shot image-goal manipulation planning research on Franka arms
  • Post-training action-conditioned world models on DROID robot trajectories

[1] [6] [13]

Known limitations
  • Code license is separate from checkpoint/data terms; binary payloads and normalization/action contract not independently validated.
  • Training config resource requests are not measured inference requirements or demonstrated robot planning.

Editorial assessment of cited source; no runtime or hardware test.

[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] · Editorial assessment

Safety and limitations
  • Inspected src/hub/backbones.py sets the checkpoint base URL to a local HTTP server on port 8300 with the public URL commented out; reconcile before using torch.hub.load.
  • Code is MIT (with Apache-2.0 preprocessing), and the separate base ViT-g HF card declares Apache-2.0, but an artifact-specific license grant for vjepa2-ac-vitg.pt remains unestablished.
  • Training config requests (4 nodes, 8 tasks/node, 220G mem_per_gpu) are workload requests, not measured inference minimums.
  • No weight deserialization, planning rollout, or physical robot testing was performed by Open Source Robotics.

[2] [3] [6] [8] [10]

Compute & fine-tuning

Inference compute
  • Documented hardware: NVIDIA CUDA GPU strongly recommended by README (training config requests 4 nodes x 8 tasks, mem_per_gpu 220G)
  • Edge / consumer capable: No

11.76 GB ViT-g + 24-layer predictor checkpoint; minimum inference VRAM and latency_ms are not established.

[1] [6] [12] · Editorial assessment

Fine-tuning

[4] [6]

Host computer

CUDA-enabled PyTorch strongly recommended by README; droid-256px-8f.yaml training config requests 4 nodes x 8 tasks with mem_per_gpu 220G (workload config, not a measured inference minimum).

[1] [6]

Operating system
  • Linux

Inferred from CUDA/PyTorch and distributed training config.

[1] [6] · Editorial assessment

Dependencies
  • README full-source Python3.12 environment; PyTorch,timm,einops for Hub route
  • Named ViT-g action-conditioned checkpoint and matched DROID trajectory/configuration; no minimum GPU capacity established

[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]

Setup & usage

Open the upstream getting-started guide [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]

  • Inspect vjepa2_ac_vit_giant Hub entry and the action-conditioned predictor before selecting a checkpoint.
  • For adaptation inspect droid-256px-8f.yaml and replace dataset/pretrain paths; keep video and robot action conventions matched.

Editorial assessment of cited source; no runtime or hardware test.

[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] · Editorial assessment

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
Zero-shot Franka manipulation with image goals in two labs (arXiv:2506.09985)Upstream paper reports zero-shot pick-and-place on Franka arms in two labs without environment-specific data collection, task-specific training, or rewardUpstream-reported results in arXiv:2506.09985; OSR has not downloaded weights, run planning, or evaluated physical robots.

[1] [13]

Project notes

Checkpoint, loader discrepancy, and post-training config

The README lists the public PyTorch checkpoint vjepa2-ac-vitg.pt (11,760,743,310 bytes / 11.76 GB, HTTP 200 verified via HEAD on 2026-11-21 without downloading the binary; ETag 12c945a28cc4f447e728fd2ba810607c-1402) and PyTorch Hub entry vjepa2_ac_vit_giant. At pinned commit 204698b45b3712590f06245fbfba32d3be539812, hubconf.py exports that entry, but src/hub/backbones.py points its checkpoint base URL to a local HTTP server on port 8300 with the public URL commented out—so users must reconcile that loader configuration before relying on torch.hub.load. Full-source setup uses Python 3.12 with PyTorch, timm, einops, and strongly recommended CUDA support. [1] [7] [8] [11] [12]

For post-training, configs/train/vitg16/droid-256px-8f.yaml configures app/vjepa_droid/train.py with 256×256 crops, 8-frame clips, and placeholder paths for DROID trajectory CSVs and a pretrained vitg.pt checkpoint. Its Slurm-style workload config requests four nodes, eight tasks per node, and mem_per_gpu: 220G; those are training workload requests rather than measured inference minimums. Minimum inference VRAM and latency remain unestablished in this review, and adapting new robot trajectories requires checking action/state conventions and normalization directly against src/models/ac_predictor.py and app/vjepa_droid/train.py—a contract that has not been independently validated here. [1] [4] [5] [6] [12]

Upstream-reported evaluations and unverified execution

According to arXiv:2506.09985, the base V-JEPA 2 encoder was pretrained on over 1 million hours of internet video, and V-JEPA 2-AC was post-trained on fewer than 62 hours of unlabeled DROID robot videos, then deployed zero-shot on Franka arms in two labs for pick-and-place via image-goal planning without task-specific training or reward. All evaluation results are strictly upstream-reported; Open Source Robotics has not downloaded weight binaries, run post-training or inference, or tested robot planning. [1] [13]

Separate code, checkpoint, and dataset terms

The facebookresearch/vjepa2 repository is licensed under MIT, with selected preprocessing utilities under Apache-2.0 (APACHE-LICENSE), and was last pushed 2026-10-29. While Meta's release blog describes code and checkpoints as available for commercial and research applications and the separate base facebook/vjepa2-vitg-fpc64-256 Hugging Face card declares Apache-2.0, an artifact-specific license notice for vjepa2-ac-vitg.pt itself remains unestablished in the inspected files, and DROID trajectory data must be obtained separately under its own terms. [2] [3] [9] [10] [14]

What we checked

Pinned code/document inspection only; no weight/data download, environment, post-training, robot planning or physical test.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Oct 29, 2026 [14]
Latest release
Not established
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
GitHub facebookresearch/vjepa2@204698b45b3712590f06245fbfba32d3be539812; checkpoint vjepa2-ac-vitg.pt (ETag 12c945a28cc4f447e728fd2ba810607c-1402) · software repository

Separate license matrix

Narrow robotics-facing2-AC implementation and source research; generic V-JEPA2/2.1 video features alone are not the selected contribution.

LayerAvailability / license
Codeavailable · MIT majority code; selected preprocessing Apache-2.0
Code notices do not automatically license downloaded model weights, DROID trajectories or unrelated assets. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]
Model weightsavailable · License not fully assessed
Official checkpoint vjepa2-ac-vitg.pt verified via HTTP HEAD (200, 11.76 GB, ETag 12c945a28cc4f447e728fd2ba810607c-1402) without binary download. Meta blog describes code and checkpoints as available for commercial/research use, and base ViT-g HF card declares Apache-2.0, but an artifact-specific license notice for the AC checkpoint itself remains unestablished. [1] [9] [10] [11] [12]
Training datasetsunavailable · License not fully assessed
Base encoder pretrained on >1M hours of internet video (not redistributed); AC post-training uses <62 hours of DROID robot trajectories supplied externally via placeholder CSV paths in droid-256px-8f.yaml. [6] [13]
Documentationavailable · MIT majority code; selected preprocessing Apache-2.0
Code notices do not automatically license downloaded model weights, DROID trajectories or unrelated assets. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Migrated from 0.7.0-draft.5 project to 0.8.0-draft.1 model; added verified 11.76 GB checkpoint metadata, ViT-g + 24-layer AC predictor architecture, and upstream-reported Franka benchmarks.