R3M
Robotics modelFrozen ResNet visual representation pre-trained on Ego4D human videos and language annotations for data-efficient robotic manipulation policy learning. [1] [5]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Robotic manipulation researchers comparing frozen visual encoders; Practitioners reproducing baseline visual representations for behavior cloning [1] [5]
- Embodiments
- Embodiment-agnostic visual representation backbone (upstream paper evaluated on 12 simulated manipulation tasks and… [5]
- Source rights
- Some licenses are unresolved. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- License
- MIT · license not fully assessed for model weights · Details by part
Overview
R3M (facebookresearch/r3m, arXiv:2203.12601) is a frozen visual representation backbone pre-trained on Ego4D human video clips and natural-language annotations using time-contrastive learning, video-language alignment, and L1 sparsity regularization. The GitHub repository is archived (archived: true) at pinned commit b2334e726887fa0206962d7984c69c5fb09cceab (last commit August 2022, last push March 2023). [1] [3] [5]
Model card summary
- Architecture
ResNet visual encoder (ResNet-18, ResNet-34, or ResNet-50) pre-trained on Ego4D video clips with time-contrastive learning (TCN), video-language alignment, and L1/L2 sparsity penalties; load_r3m() wraps the encoder in torch.nn.DataParallel and removes lang_enc/lang_rew weights for visual-only downstream use.
- Base model
ResNet-18, ResNet-34, and ResNet-50 visual backbones (codebase adapted in part from DrQV2)
- Parameter count
Not explicitly stated as a scalar in README or arXiv:2203.12601 abstract; released in ResNet-18, ResNet-34, and ResNet-50 backbone variants with language head removed at load time
Exact parameter count not printed in README; Google Drive .pt payloads were not downloaded.
- Input modalities
- rgb
224x224 RGB frames at inference; pre-training also uses Ego4D natural-language clip annotations, but load_r3m() strips the language head.
- Output modalities
- video-frames
R3M is a frozen visual representation encoder outputting compact visual embeddings rather than direct robot actions; mapped to video-frames as the closest controlled-vocabulary visual modality.
- Action space
Visual representation backbone only (no direct robot action output); produces compact visual feature embeddings used as frozen inputs to downstream behavior cloning or RL policies.
- Trained / supported embodiments
- Embodiment-agnostic visual representation backbone (upstream paper evaluated on 12 simulated manipulation tasks and a real Franka Emika Panda arm)
- Training datasets
- Ego4D human video dataset parsed into 224x224 video clip frames paired with natural language annotations (manifest.csv)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| R3M ResNet-50 (r3m_50) | pytorch | Not stated | open | main / b2334e726887 |
| R3M ResNet-34 (r3m_34) | pytorch | Not stated | open | main / b2334e726887 |
| R3M ResNet-18 (r3m_18) | pytorch | Not stated | open | main / b2334e726887 |
Three primary Google Drive PyTorch checkpoints defined in r3m/__init__.py (load_r3m); payloads were not downloaded so exact file sizes are unknown. Four additional ablation checkpoints exist in load_r3m_reproduce().
Is this for you?
- Use cases
- Frozen visual feature extraction for simulated and real-world robotic manipulation behavior cloning
- Reproducing baseline pre-trained visual representation comparisons against newer encoders
- Known limitations
- GitHub repository is archived (read-only) with last push in March 2023 and no ongoing dependency maintenance.
- Pre-trained checkpoints are downloaded via gdown from individual Google Drive file IDs rather than a versioned model hub, and external weight payloads were not downloaded or verified by OSR.
- r3m/__init__.py strips the language encoder and reward head at load time (load_r3m) and contains an undefined-variable bug ('modelif' instead of 'modelid' in elif branches, raising NameError if evaluated) in load_r3m_reproduce() for r3m_nol1/r3m_nolang.
- Training from scratch requires obtaining and pre-parsing the external Ego4D dataset into 224x224 frame directories and manifest.csv.
- Safety and limitations
- Archived repository (read-only since 2023) with pinned legacy conda dependencies (r3m_base.yaml) and gdown-based Google Drive checkpoint fetches that may break or rate-limit.
- R3M is a frozen visual feature encoder only and outputs no robot control commands; downstream policy safety and hardware limits must be engineered separately.
- External Google Drive checkpoints carry no separate verified license file inside this review; Ego4D pre-training data carries its own separate dataset license.
- Open Source Robotics has not installed R3M, downloaded checkpoints, run inference, or reproduced manipulation benchmarks.
Compute & fine-tuning
- Inference compute
- Documented hardware: PyTorch CUDA GPU or CPU (r3m/__init__.py selects cuda if available else cpu; exact VRAM and latency not stated in README)
- Edge / consumer capable: No
load_r3m() supports CUDA or CPU in PyTorch; no minimum VRAM, latency, or edge benchmark is documented.
[4] · Editorial assessment
- Fine-tuning
- Supported: Yes
- Methods: pre-training from scratch via train_representation.py (Hydra + Ego4D); downstream behavior cloning on frozen representation (eval branch)
- Recipe: Upstream fine-tuning documentation
- Host computer
PyTorch CPU or CUDA GPU host for inference (r3m/__init__.py falls back to CPU if CUDA is unavailable); multi-hour GPU compute and parsed Ego4D storage required for representation pre-training.
Inferred from r3m/__init__.py device selection and README training instructions.
- Operating system
- Linux
Conda YAML environment and standard robotics research stack target Linux.
[1] · Editorial assessment
- Dependencies
- PyTorch
- Hydra and OmegaConf
- gdown (for Google Drive checkpoint downloads)
- Conda environment (r3m_base.yaml)
Setup & usage
Open the upstream getting-started guide [1]
- Create a conda environment from r3m/r3m_base.yaml (or use an existing PyTorch conda environment).
- Run pip install -e . from the repository root and verify import r3m.
- Call r3m.load_r3m('resnet50') (or 'resnet34', 'resnet18') to fetch the Google Drive model.pt and config.yaml via gdown into ~/.r3m/.
- Use the frozen encoder in eval() mode as the visual frontend for downstream behavior cloning (see the separate eval branch).
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| 12 Simulated Robot Manipulation Tasks (arXiv:2203.12601) | Improves task success by >20% compared to training from scratch and >10% compared to CLIP and MoCo. | Upstream-reported paper abstract summary; OSR has not reproduced behavior-cloning evaluations. |
| Real-World Franka Emika Panda Manipulation in Cluttered Apartment (arXiv:2203.12601) | Enables learning a range of manipulation tasks given 20 demonstrations per task. | Upstream-reported hardware evaluation in paper abstract; not independently verified by OSR. |
Project notes
Architecture, checkpoints, and code caveats
R3M provides three primary ResNet backbones—resnet50, resnet34, and resnet18—loaded via r3m.load_r3m(), which downloads model.pt and config.yaml from hardcoded Google Drive IDs using gdown. At load time, r3m/__init__.py explicitly strips the language encoder and reward head (lang_enc, lang_rew) and wraps the visual encoder in torch.nn.DataParallel so downstream policies consume compact visual embeddings from 224×224 RGB frames. [1] [4]
Because the repository is archived and fetches checkpoints via gdown from Google Drive, users face legacy dependency pinning (r3m_base.yaml) and the possible risk of Google Drive rate limiting or link rot; current Google Drive download reliability and compatibility with modern PyTorch/CUDA stacks have not been verified in this review. Separately from the primary load_r3m() path, load_r3m_reproduce() defines four reproduction/ablation IDs (r3m, r3m_noaug, r3m_nol1, and r3m_nolang): r3m and r3m_noaug are checked first, whereas execution reaching the r3m_nol1 or r3m_nolang branches hits an undefined-variable bug (modelif instead of modelid) that raises a NameError (download availability for all Google Drive IDs remains unverified). Pre-training from scratch requires obtaining and parsing the external Ego4D dataset into per-clip 224×224 frame directories and a manifest.csv index. [1] [3] [4]
Upstream-reported evaluation
In arXiv:2203.12601, the authors report that using R3M as a frozen visual encoder across 12 simulated robot manipulation tasks improves task success by over 20% compared to training from scratch and by over 10% compared to CLIP and MoCo, and enables a real Franka Emika Panda arm to learn manipulation tasks in a cluttered apartment from 20 demonstrations. All reported results are upstream-reported; Open Source Robotics has not installed R3M, downloaded the Google Drive checkpoints, run training, fine-tuning, or inference, or reproduced any benchmarks. [5]
What we checked
Source researched only; no installation, training, fine-tuning, inference, benchmark, or physical hardware test performed. No affiliate links or reused media.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Upstream last updated
- Mar 21, 2023 [3]
- Latest release
- Not established
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- archived commit snapshot; no GitHub release / b2334e726887fa0206962d7984c69c5fb09cceab · software repository
Separate license matrix
Include the MIT-licensed R3M visual representation codebase as a partially open archived robotics baseline; code is MIT, while external Google Drive checkpoints lack a verified standalone license and Ego4D pre-training data is external.
| Layer | Availability / license |
|---|---|
| Code | available · MIT Permissive MIT License in root LICENSE; repository is archived. [2] [3] |
| Model weights | available · License not fully assessed Three ResNet checkpoints are fetched via gdown from Google Drive URLs in r3m/__init__.py; payloads were not downloaded and no separate artifact-specific weight license file was verified. [1] [4] |
| Training datasets | not found · License not fully assessed Pre-trained on the external Ego4D dataset, which is not redistributed in this repository and carries its own separate access agreement. [1] [5] |
| Documentation | available · MIT README and example scripts in archived repository. [1] [2] |
Licensing and dataset separation
The repository code and documentation carry the permissive MIT License (LICENSE). However, the pre-trained checkpoints are distributed externally via Google Drive without a separate verified license file inside this review, and the underlying Ego4D pre-training dataset is governed by its own separate dataset license and access terms. [1] [2] [4] [5]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 R3M README
Installation, load_r3m() usage, Ego4D training command, eval branch pointer, and MIT license statement.Revision, date & where cited
facebookresearch/r3m contributors · github.com · revision
b2334e7· Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 4 more - S002 R3M MIT License
MIT License covering repository software and documentation.Revision, date & where cited
Facebook, Inc. and its affiliates · github.com · revision
b2334e7· Accessed Nov 21, 2026
Used for: Is this for you?, Separate license matrix - S003 GitHub repository metadata and pinned commit
Archived repository (archived=true); last commit 2022-08-18, last push 2023-03-21; no GitHub releases.Revision, date & where cited
facebookresearch/r3m / GitHub · github.com · revision
b2334e7· main / · Accessed Nov 21, 2026
Used for: Overview, Is this for you?, Architecture, checkpoints, and code caveats, What we checked and 1 more - S004 R3M checkpoint loader (r3m/__init__.py)
Documents exact Google Drive gdown URLs for resnet50, resnet34, and resnet18 checkpoints, language head stripping at load time, and ablation loader.Revision, date & where cited
facebookresearch/r3m contributors · github.com · revision
b2334e7· Accessed Nov 21, 2026
Used for: Model card summary, Checkpoints, Is this for you?, Compute & fine-tuning and 3 more - S005 R3M: A Universal Visual Representation for Robot Manipulation (arXiv:2203.12601)
Pre-trained on Ego4D using time-contrastive learning, video-language alignment, and L1 sparsity; reports >20% gain over scratch and >10% over CLIP/MoCo across 12 simulated manipulation tasks and real Franka Emika Panda tasks from 20 demos.Revision, date & where cited
Stanford University & Meta AI (arXiv:2203.12601v3) · arxiv.org · arXiv:2203.12601v3 abstract · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Upstream-reported benchmarks and 2 more
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile for R3M (archived MIT repo, 3 ResNet checkpoints).