Open Source RoboticsRobots, software & learning

VC-1 (Visual Cortex)

Robotics model

Masked Auto-Encoder (MAE) Vision Transformer visual backbone (ViT-L and ViT-B) pre-trained on 5.62M egocentric video frames and ImageNet for Embodied AI tasks. [1] [2] [5]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Embodied AI researchers evaluating pre-trained visual representations on CortexBench; Robot learning teams studying MAE visual backbones under non-commercial terms [1] [2] [3]
Parameters
Not explicitly stated as a scalar in README or MODEL_CARD.md [2] [6]
Embodiments
Embodiment-agnostic visual representation backbone (evaluated across Adroit, Meta-World, DMControl, TriFinger,… [2] [5]
Source rights
Named licenses are documented by part. Includes noncommercial terms. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware

Requirements · Research details and dates

On this page

Overview

VC-1 (facebookresearch/eai-vc, arXiv:2303.18240) is a Masked Auto-Encoder (MAE) Vision Transformer pre-trained on approximately 5.62 million images in total (EgoMNI: 4,340,820 egocentric video frames from over 4,000 hours of video plus 1,281,167 ImageNet images), released alongside the 17-task CortexBench Embodied AI evaluation suite. At pinned commit 76fe35e87b1937168f1ec4b236e863451883eaf3 (last commit July 2023, last push May 2024), the repository provides the vc_models loader, a detailed MODEL_CARD.md, and direct public downloads for ViT-L and ViT-B checkpoints. [1] [2] [4] [5]

Model card summary

Architecture

Vision Transformer pre-trained for 182 epochs with Masked Auto-Encoding (MAE): VC-1 uses ViT-L (16x16 patches, 24 layers, 16 heads, 1024 embedding dim, MLP ratio 4, CLS token) and VC-1-base uses ViT-B (768 embedding dim).

[1] [2]

Parameter count

Not explicitly stated as a scalar in README or MODEL_CARD.md; ViT-L (24 layers, 1024 dim, 16 heads; 1.213 GB .pth) and ViT-B (768 dim; 0.343 GB .pth)

Architecture dimensions from MODEL_CARD.md and file sizes from HTTP HEAD checks; weights were not loaded to count parameters.

[2] [6] · Editorial assessment

Input modalities
rgb

RGB images transformed to 224x224x3 by model_transforms.

[1] [2]

Output modalities
video-frames

VC-1 is a visual encoder outputting 1024-D (ViT-L) or 768-D (ViT-B) frame embeddings rather than robot actions; mapped to video-frames as the closest controlled-vocabulary visual modality.

[1] [2] · Editorial assessment

Action space

Visual foundation backbone only (no direct robot action output); outputs 1024-D (ViT-L) or 768-D (ViT-B) image embeddings consumed by downstream embodied AI policies.

[1] [2]

Trained / supported embodiments
  • Embodiment-agnostic visual representation backbone (evaluated across Adroit, Meta-World, DMControl, TriFinger, Habitat, Habitat 2.0, and real-world hardware in upstream paper)

[2] [5]

Training datasets
  • EgoMNI assembled dataset (5,621,987 frames): Ego4D (2,790,520), 100DOH (99,899), Something-Something v2 (315,115), Epic Kitchens (332,757), OpenHouse24 (499,442), RealEstate10K (303,087), and ImageNet (1,281,167)

[2]

Interfaces
  • Python API (vc_models.models.vit.model_utils.load_model)
  • Hydra/YAML encoder config plugins for CortexBench

[1]

Checkpoints

CheckpointFormatSizeAccessPinned revision
VC-1 (ViT-L)pytorch1.213 GBopenv1.0 / 2023-07-20
VC-1-base (ViT-B)pytorch0.343 GBopenv1.0 / 2023-07-20

Both direct HTTPS .pth checkpoints verified HTTP 200 with exact Content-Length on 2026-11-21.

[1] [2] [6]

Is this for you?

Use cases
  • Extracting 1024-D (ViT-L) or 768-D (ViT-B) visual embeddings for downstream manipulation, navigation, and locomotion policies
  • Domain- or task-specific adaptation of a pre-trained MAE visual encoder to embodied control benchmarks
  • Benchmarking custom visual encoders across the 17 tasks in CortexBench

[1] [2] [5]

Known limitations
  • Primary repository and model license is CC-BY-NC-4.0 (non-commercial only), with additional CC-BY-NC-SA-3.0 US and Matterport3D/Gibson Terms of Use on derived navigation datasets and models.
  • Upstream paper explicitly notes that frozen VC-1 does not universally dominate all CortexBench tasks without task- or domain-specific adaptation.
  • MODEL_CARD.md notes that ViT-L size may pose challenges on resource-constrained devices and that simulation benchmark rankings may differ on real robots.
  • Pre-training required over 10,000 GPU-hours across 7 external video/image datasets that must be obtained separately.

[1] [2] [3] [5]

Safety and limitations
  • CC-BY-NC-4.0 prohibits commercial deployment; third-party benchmark components carry separate BSD-3-Clause, Apache-2.0, MIT, and Matterport3D/Gibson terms.
  • VC-1 is a visual feature encoder only and produces no direct robot actions; downstream policy safety and physical robot validation remain external.
  • Upstream MODEL_CARD.md warns that simulated CortexBench tasks are proxies and real-world robot performance can diverge from benchmark rankings.
  • Open Source Robotics has not installed vc_models, run inference, or reproduced CortexBench results.

[1] [2] [3]

Compute & fine-tuning

Inference compute
  • Documented hardware: PyTorch GPU/CPU (exact minimum VRAM and latency not stated; MODEL_CARD.md notes ViT-L size may challenge resource-constrained devices)
  • Edge / consumer capable: No

MODEL_CARD.md flags large model size for resource-constrained devices; no VRAM or latency benchmarks published in README/MODEL_CARD.

[2] · Editorial assessment

Fine-tuning

[1] [5]

Host computer

PyTorch GPU recommended for ViT-L inference and CortexBench policy training; upstream MAE pre-training consumed over 10,000 GPU-hours.

[2] [5]

Operating system
  • Linux

Conda environment and Habitat/MuJoCo CortexBench stack target Linux.

[1] · Editorial assessment

Dependencies
  • Python 3.8
  • PyTorch and Conda (environment.yml)
  • Optional CortexBench simulators (Habitat Lab, MuJoCo/mj_envs, dmc2gym, trifinger_simulation)

[1] [2]

Skills
  • Python 3.8, PyTorch, and Conda
  • Embodied AI simulation benchmarks (Habitat, MuJoCo, Meta-World)

Inferred from README badges and CortexBench task list.

[1] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1] · Model card / documentation [2]

  • Follow INSTALLATION.md with Python 3.8 and conda (environment.yml) to install the vc_models package.
  • Load VC-1 via vc_models.models.vit.model_utils.load_model(model_utils.VC1_LARGE_NAME) or VC1_BASE_NAME.
  • Apply the returned model_transforms to resize/crop inputs to Bx3x224x224 and run model(transformed_img) to obtain embeddings.
  • Optionally configure CortexBench task submodules under cortexbench/ to reproduce benchmark evaluations.

[1] [2]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
CortexBench (17 tasks across 7 benchmarks: Adroit, Meta-World, DMControl, TriFinger, ObjectNav, ImageNav, Mobile Pick)VC-1 (ViT-L, Ego4D+MNI) Mean Success 68.7 (Mean Rank 2.4): Adroit 59.3±5.2, Meta-World 88.8±2.2, DMControl 66.9±1.4, TriFinger 71.7±0.4, ObjectNav 60.3±1.1, ImageNav 70.3±0.7, Mobile Pick 63.2±2.2.Upstream-reported frozen PVR scores from MODEL_CARD.md; paper notes adaptation yields further gains. OSR has not reproduced CortexBench.

[2] [5]

Project notes

Architecture, checkpoints, and setup boundaries

VC-1 is a visual representation backbone—not a direct action policy—that transforms RGB images to 224×224×3 and outputs 1024-dimensional embeddings (VC-1, ViT-L with 24 layers and 16 heads) or 768-dimensional embeddings (VC-1-base, ViT-B). Both PyTorch checkpoints are hosted directly on Meta's public file server and verified reachable via HTTP HEAD: vc1_vitl.pth (1.213 GB) and vc1_vitb.pth (0.343 GB). Pre-training the released models required over 10,000 GPU-hours on the assembled EgoMNI dataset. [1] [2] [5] [6]

Installation targets Python 3.8 via environment.yml and INSTALLATION.md. While vc_models can be installed standalone for feature extraction, reproducing CortexBench requires setting up multiple third-party simulation environments (Habitat, MuJoCo, Meta-World, TriFinger). Upstream's model card notes that ViT-L's size may challenge resource-constrained devices and that simulated benchmark rankings serve only as proxies for real-robot deployment. Minimum GPU VRAM and inference latency are not established in the supplied README or model card, and compatibility with modern Python/PyTorch environments has not been independently tested. [1] [2]

Upstream-reported CortexBench results

In MODEL_CARD.md and arXiv:2303.18240, frozen VC-1 (ViT-L, Ego4D+MNI) achieves a Mean Success of 68.7 and Mean Rank of 2.4 across the 7 CortexBench benchmark groups (Adroit 59.3±5.2, Meta-World 88.8±2.2, DMControl 66.9±1.4, TriFinger 71.7±0.4, ObjectNav 60.3±1.1, ImageNav 70.3±0.7, and Mobile Pick 63.2±2.2). The paper explicitly notes that neither VC-1 nor prior frozen visual representations universally dominate every task without task- or domain-specific adaptation. All results are upstream-reported; Open Source Robotics has not installed vc_models, run training, fine-tuning, or inference, or reproduced CortexBench. [2] [5]

What we checked

Source researched only; no installation, training, fine-tuning, inference, benchmark, or physical hardware test performed. No affiliate links or reused media.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
May 1, 2024 [4]
Latest release
Not established
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
commit snapshot; no GitHub release / 76fe35e87b1937168f1ec4b236e863451883eaf3 · software repository

Separate license matrix

Include the inspectable VC-1 MAE visual backbone and verified public checkpoints under a non-commercial research scope; clearly distinguish CC-BY-NC-4.0 terms and separate third-party benchmark/dataset licenses.

LayerAvailability / license
Codeavailable · CC-BY-NC-4.0 (with BSD-3-Clause, Apache-2.0, and MIT third-party components)
Majority of Visual Cortex and CortexBench code is CC-BY-NC-4.0; README lists separate licenses for trifinger_simulation (BSD-3-Clause), mj_envs/mjrl (Apache-2.0), and Habitat Lab/dmc2gym/mujoco-py (MIT). [1] [3]
Model weightsavailable · CC-BY-NC-4.0
Direct Meta public HTTPS downloads verified HTTP 200 (vc1_vitl.pth 1.213 GB, vc1_vitb.pth 0.343 GB); subject to repository CC-BY-NC-4.0 non-commercial terms. [1] [3] [6]
Training datasetsnot found · License not fully assessed
Pre-training uses 7 external datasets (Ego4D, 100DOH, SS-v2, Epic Kitchens, OpenHouse24, RealEstate10K, ImageNet) not redistributed here; Matterport3D/Gibson task datasets carry CC-BY-NC-SA-3.0 US and upstream Terms of Use. [1] [2]
Documentationavailable · CC-BY-NC-4.0
Includes README.md, MODEL_CARD.md, INSTALLATION.md, and tutorial notebook. [2] [3]

Non-commercial licensing and dataset terms

The root LICENSE is Creative Commons Attribution-NonCommercial 4.0 International (CC-BY-NC-4.0), barring commercial use of the main codebase and VC-1 checkpoints. The README further documents separate third-party licenses for bundled benchmark components (BSD-3-Clause, Apache-2.0, MIT) and notes that Matterport3D and Gibson task datasets and derived policy models remain subject to CC-BY-NC-SA-3.0 US and their respective upstream Terms of Use. [1] [3]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile for VC-1 with 2 verified checkpoints and CortexBench table from MODEL_CARD.md.