V-JEPA 2-AC research workflow

Project notes

Inspect a named checkpoint and loader

The README names the ViT-g action-conditioned checkpoint vjepa2-ac-vitg.pt and the Hub entry vjepa2_ac_vit_giant. The inspected hubconf.py exports that entry, but backbones.py actively sets its checkpoint base URL to a local HTTP server on port 8300, with the public endpoint commented out. That loader configuration differs from the README's public download: reconcile the selected revision/configuration before treating the Hub route as a reproduction starting point. A public-download availability check does not validate this local-server loader. The README asks for PyTorch, timm and einops, strongly recommends CUDA-enabled PyTorch, and adds Jupyter/SciPy for an energy-landscape notebook using a laboratory robot trajectory. The full-source installation example creates a Python 3.12 environment. None of these routes was installed or executed here. [1] [7] [8]

The checkpoint is a documented public download, not a binary inspected by this review. The official announcement describes code and checkpoints as available for commercial and research applications. The README-linked Hugging Face collection leads to a base ViT-g card with an Apache-2.0 declaration; that card describes the generic video encoder, not this AC artifact. An artifact-specific AC license and the underlying trajectory terms remain unestablished here. Do not transfer either code MIT or the base-card declaration to a different download. [9] [10] No weight deserialization, prediction quality or robot-action normalization was validated. [1] [2] [3]

Treat post-training configuration as a workload example

The selected droid-256px-8f.yaml chooses app vjepa_droid, 256-pixel crops, eight-frame clips and a left-camera video path. It contains placeholders for a trajectory-directory CSV and a pretrained vitg.pt checkpoint. Those files and the recorded robot action conventions must be supplied and matched before attempting adaptation; a video model alone does not define a usable robot command. [6]

That configuration requests four nodes, eight tasks per node and mem_per_gpu 220G. These are configuration resource requests for a training workload, not an independently measured minimum GPU requirement or an inference budget. The action-conditioned training/predictor source provides implementation to investigate; no distributed training, dataset ingestion or physical planning was tested here. [4] [5] [6]

The root code notice is MIT, with selected preprocessing utilities separately identified as Apache-2.0. The repository's general robot-manipulation claims remain upstream reports. Begin with the named offline prediction/source workflow and resolve checkpoint/data rights and action conventions before considering any hardware integration. [1] [2] [3]