VC-1 (Visual Cortex)
Robotics modelMasked Auto-Encoder (MAE) Vision Transformer visual backbone (ViT-L and ViT-B) pre-trained on 5.62M egocentric video frames and ImageNet for Embodied AI tasks. [1] [2] [5]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Embodied AI researchers evaluating pre-trained visual representations on CortexBench; Robot learning teams studying MAE visual backbones under non-commercial terms [1] [2] [3]
- Embodiments
- Embodiment-agnostic visual representation backbone (evaluated across Adroit, Meta-World, DMControl, TriFinger,… [2] [5]
- Source rights
- Named licenses are documented by part. Includes noncommercial terms. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- License
- CC-BY-NC-4.0 (with BSD-3-Clause, Apache-2.0, and MIT third-party components) (noncommercial), CC-BY-NC-4.0 (noncommercial) · Details by part
Overview
VC-1 (facebookresearch/eai-vc, arXiv:2303.18240) is a Masked Auto-Encoder (MAE) Vision Transformer pre-trained on approximately 5.62 million images in total (EgoMNI: 4,340,820 egocentric video frames from over 4,000 hours of video plus 1,281,167 ImageNet images), released alongside the 17-task CortexBench Embodied AI evaluation suite. At pinned commit 76fe35e87b1937168f1ec4b236e863451883eaf3 (last commit July 2023, last push May 2024), the repository provides the vc_models loader, a detailed MODEL_CARD.md, and direct public downloads for ViT-L and ViT-B checkpoints. [1] [2] [4] [5]
Model card summary
- Architecture
Vision Transformer pre-trained for 182 epochs with Masked Auto-Encoding (MAE): VC-1 uses ViT-L (16x16 patches, 24 layers, 16 heads, 1024 embedding dim, MLP ratio 4, CLS token) and VC-1-base uses ViT-B (768 embedding dim).
- Parameter count
Not explicitly stated as a scalar in README or MODEL_CARD.md; ViT-L (24 layers, 1024 dim, 16 heads; 1.213 GB .pth) and ViT-B (768 dim; 0.343 GB .pth)
Architecture dimensions from MODEL_CARD.md and file sizes from HTTP HEAD checks; weights were not loaded to count parameters.
- Output modalities
- video-frames
VC-1 is a visual encoder outputting 1024-D (ViT-L) or 768-D (ViT-B) frame embeddings rather than robot actions; mapped to video-frames as the closest controlled-vocabulary visual modality.
- Action space
Visual foundation backbone only (no direct robot action output); outputs 1024-D (ViT-L) or 768-D (ViT-B) image embeddings consumed by downstream embodied AI policies.
- Trained / supported embodiments
- Embodiment-agnostic visual representation backbone (evaluated across Adroit, Meta-World, DMControl, TriFinger, Habitat, Habitat 2.0, and real-world hardware in upstream paper)
- Training datasets
- EgoMNI assembled dataset (5,621,987 frames): Ego4D (2,790,520), 100DOH (99,899), Something-Something v2 (315,115), Epic Kitchens (332,757), OpenHouse24 (499,442), RealEstate10K (303,087), and ImageNet (1,281,167)
- Interfaces
- Python API (vc_models.models.vit.model_utils.load_model)
- Hydra/YAML encoder config plugins for CortexBench
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| VC-1 (ViT-L) | pytorch | 1.213 GB | open | v1.0 / 2023-07-20 |
| VC-1-base (ViT-B) | pytorch | 0.343 GB | open | v1.0 / 2023-07-20 |
Both direct HTTPS .pth checkpoints verified HTTP 200 with exact Content-Length on 2026-11-21.
Is this for you?
- Use cases
- Extracting 1024-D (ViT-L) or 768-D (ViT-B) visual embeddings for downstream manipulation, navigation, and locomotion policies
- Domain- or task-specific adaptation of a pre-trained MAE visual encoder to embodied control benchmarks
- Benchmarking custom visual encoders across the 17 tasks in CortexBench
- Known limitations
- Primary repository and model license is CC-BY-NC-4.0 (non-commercial only), with additional CC-BY-NC-SA-3.0 US and Matterport3D/Gibson Terms of Use on derived navigation datasets and models.
- Upstream paper explicitly notes that frozen VC-1 does not universally dominate all CortexBench tasks without task- or domain-specific adaptation.
- MODEL_CARD.md notes that ViT-L size may pose challenges on resource-constrained devices and that simulation benchmark rankings may differ on real robots.
- Pre-training required over 10,000 GPU-hours across 7 external video/image datasets that must be obtained separately.
- Safety and limitations
- CC-BY-NC-4.0 prohibits commercial deployment; third-party benchmark components carry separate BSD-3-Clause, Apache-2.0, MIT, and Matterport3D/Gibson terms.
- VC-1 is a visual feature encoder only and produces no direct robot actions; downstream policy safety and physical robot validation remain external.
- Upstream MODEL_CARD.md warns that simulated CortexBench tasks are proxies and real-world robot performance can diverge from benchmark rankings.
- Open Source Robotics has not installed vc_models, run inference, or reproduced CortexBench results.
Compute & fine-tuning
- Inference compute
- Documented hardware: PyTorch GPU/CPU (exact minimum VRAM and latency not stated; MODEL_CARD.md notes ViT-L size may challenge resource-constrained devices)
- Edge / consumer capable: No
MODEL_CARD.md flags large model size for resource-constrained devices; no VRAM or latency benchmarks published in README/MODEL_CARD.
[2] · Editorial assessment
- Fine-tuning
- Supported: Yes
- Methods: task- or domain-specific MAE / downstream policy adaptation reported in paper and CortexBench
- Recipe: Upstream fine-tuning documentation
- Host computer
PyTorch GPU recommended for ViT-L inference and CortexBench policy training; upstream MAE pre-training consumed over 10,000 GPU-hours.
- Operating system
- Linux
Conda environment and Habitat/MuJoCo CortexBench stack target Linux.
[1] · Editorial assessment
- Dependencies
- Python 3.8
- PyTorch and Conda (environment.yml)
- Optional CortexBench simulators (Habitat Lab, MuJoCo/mj_envs, dmc2gym, trifinger_simulation)
- Skills
- Python 3.8, PyTorch, and Conda
- Embodied AI simulation benchmarks (Habitat, MuJoCo, Meta-World)
Inferred from README badges and CortexBench task list.
[1] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [2]
- Follow INSTALLATION.md with Python 3.8 and conda (environment.yml) to install the vc_models package.
- Load VC-1 via vc_models.models.vit.model_utils.load_model(model_utils.VC1_LARGE_NAME) or VC1_BASE_NAME.
- Apply the returned model_transforms to resize/crop inputs to Bx3x224x224 and run model(transformed_img) to obtain embeddings.
- Optionally configure CortexBench task submodules under cortexbench/ to reproduce benchmark evaluations.
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| CortexBench (17 tasks across 7 benchmarks: Adroit, Meta-World, DMControl, TriFinger, ObjectNav, ImageNav, Mobile Pick) | VC-1 (ViT-L, Ego4D+MNI) Mean Success 68.7 (Mean Rank 2.4): Adroit 59.3±5.2, Meta-World 88.8±2.2, DMControl 66.9±1.4, TriFinger 71.7±0.4, ObjectNav 60.3±1.1, ImageNav 70.3±0.7, Mobile Pick 63.2±2.2. | Upstream-reported frozen PVR scores from MODEL_CARD.md; paper notes adaptation yields further gains. OSR has not reproduced CortexBench. |
Project notes
Architecture, checkpoints, and setup boundaries
VC-1 is a visual representation backbone—not a direct action policy—that transforms RGB images to 224×224×3 and outputs 1024-dimensional embeddings (VC-1, ViT-L with 24 layers and 16 heads) or 768-dimensional embeddings (VC-1-base, ViT-B). Both PyTorch checkpoints are hosted directly on Meta's public file server and verified reachable via HTTP HEAD: vc1_vitl.pth (1.213 GB) and vc1_vitb.pth (0.343 GB). Pre-training the released models required over 10,000 GPU-hours on the assembled EgoMNI dataset. [1] [2] [5] [6]
Installation targets Python 3.8 via environment.yml and INSTALLATION.md. While vc_models can be installed standalone for feature extraction, reproducing CortexBench requires setting up multiple third-party simulation environments (Habitat, MuJoCo, Meta-World, TriFinger). Upstream's model card notes that ViT-L's size may challenge resource-constrained devices and that simulated benchmark rankings serve only as proxies for real-robot deployment. Minimum GPU VRAM and inference latency are not established in the supplied README or model card, and compatibility with modern Python/PyTorch environments has not been independently tested. [1] [2]
Upstream-reported CortexBench results
In MODEL_CARD.md and arXiv:2303.18240, frozen VC-1 (ViT-L, Ego4D+MNI) achieves a Mean Success of 68.7 and Mean Rank of 2.4 across the 7 CortexBench benchmark groups (Adroit 59.3±5.2, Meta-World 88.8±2.2, DMControl 66.9±1.4, TriFinger 71.7±0.4, ObjectNav 60.3±1.1, ImageNav 70.3±0.7, and Mobile Pick 63.2±2.2). The paper explicitly notes that neither VC-1 nor prior frozen visual representations universally dominate every task without task- or domain-specific adaptation. All results are upstream-reported; Open Source Robotics has not installed vc_models, run training, fine-tuning, or inference, or reproduced CortexBench. [2] [5]
What we checked
Source researched only; no installation, training, fine-tuning, inference, benchmark, or physical hardware test performed. No affiliate links or reused media.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Upstream last updated
- May 1, 2024 [4]
- Latest release
- Not established
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- commit snapshot; no GitHub release / 76fe35e87b1937168f1ec4b236e863451883eaf3 · software repository
Separate license matrix
Include the inspectable VC-1 MAE visual backbone and verified public checkpoints under a non-commercial research scope; clearly distinguish CC-BY-NC-4.0 terms and separate third-party benchmark/dataset licenses.
| Layer | Availability / license |
|---|---|
| Code | available · CC-BY-NC-4.0 (with BSD-3-Clause, Apache-2.0, and MIT third-party components) Majority of Visual Cortex and CortexBench code is CC-BY-NC-4.0; README lists separate licenses for trifinger_simulation (BSD-3-Clause), mj_envs/mjrl (Apache-2.0), and Habitat Lab/dmc2gym/mujoco-py (MIT). [1] [3] |
| Model weights | available · CC-BY-NC-4.0 Direct Meta public HTTPS downloads verified HTTP 200 (vc1_vitl.pth 1.213 GB, vc1_vitb.pth 0.343 GB); subject to repository CC-BY-NC-4.0 non-commercial terms. [1] [3] [6] |
| Training datasets | not found · License not fully assessed Pre-training uses 7 external datasets (Ego4D, 100DOH, SS-v2, Epic Kitchens, OpenHouse24, RealEstate10K, ImageNet) not redistributed here; Matterport3D/Gibson task datasets carry CC-BY-NC-SA-3.0 US and upstream Terms of Use. [1] [2] |
| Documentation | available · CC-BY-NC-4.0 Includes README.md, MODEL_CARD.md, INSTALLATION.md, and tutorial notebook. [2] [3] |
Non-commercial licensing and dataset terms
The root LICENSE is Creative Commons Attribution-NonCommercial 4.0 International (CC-BY-NC-4.0), barring commercial use of the main codebase and VC-1 checkpoints. The README further documents separate third-party licenses for bundled benchmark components (BSD-3-Clause, Apache-2.0, MIT) and notes that Matterport3D and Gibson task datasets and derived policy models remain subject to CC-BY-NC-SA-3.0 US and their respective upstream Terms of Use. [1] [3]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 Visual Cortex and CortexBench README
VC-1 checkpoints, loading API, CortexBench structure, and mixed licensing notices.Revision, date & where cited
facebookresearch/eai-vc contributors · github.com · revision
76fe35e· Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 4 more - S002 VC-1 Model Card
Detailed ViT-L architecture, EgoMNI 5.62M-frame dataset breakdown, CortexBench 17-task performance table, and limitations.Revision, date & where cited
FAIR / facebookresearch/eai-vc · github.com · revision
76fe35e· Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 6 more - S003 CC-BY-NC-4.0 License
Creative Commons Attribution-NonCommercial 4.0 International license.Revision, date & where cited
facebookresearch/eai-vc · github.com · revision
76fe35e· Accessed Nov 21, 2026
Used for: Is this for you?, Separate license matrix - S004 GitHub repository metadata and pinned commit
Last commit 2023-07-19, last push 2024-05-01; no GitHub releases; not archived.Revision, date & where cited
facebookresearch/eai-vc / GitHub · github.com · revision
76fe35e· main / · Accessed Nov 21, 2026
Used for: Overview, What we checked - S005 Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence? (arXiv:2303.18240)
Empirical study of PVRs on CortexBench (17 tasks), MAE pre-training (>10,000 GPU-hours) on EgoMNI, adaptation gains, and real-world hardware evaluation.Revision, date & where cited
Meta AI / FAIR & academic collaborators (arXiv:2303.18240v2) · arxiv.org · arXiv:2303.18240v2 abstract · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Compute & fine-tuning and 3 more - S006 Meta public checkpoint HEAD verification
Verified HTTP 200 for vc1_vitl.pth (1,213,303,140 bytes, 1.213 GB, Last-Modified 2023-07-20) and vc1_vitb.pth (343,240,723 bytes, 0.343 GB, Last-Modified 2023-07-20).Revision, date & where cited
Meta / fbaipublicfiles · dl.fbaipublicfiles.com · HTTP HEAD 2026-11-21 · Accessed Nov 21, 2026
Used for: Model card summary, Checkpoints, Architecture, checkpoints, and setup boundaries, Separate license matrix
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile for VC-1 with 2 verified checkpoints and CortexBench table from MODEL_CARD.md.