Open Source RoboticsRobots, software & learning

R3M

Robotics model

Frozen ResNet visual representation pre-trained on Ego4D human videos and language annotations for data-efficient robotic manipulation policy learning. [1] [5]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Robotic manipulation researchers comparing frozen visual encoders; Practitioners reproducing baseline visual representations for behavior cloning [1] [5]
Parameters
Not explicitly stated as a scalar in README or arXiv:2203.12601 abstract [1] [4]
Embodiments
Embodiment-agnostic visual representation backbone (upstream paper evaluated on 12 simulated manipulation tasks and… [5]
Source rights
Some licenses are unresolved. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware
License
MIT · license not fully assessed for model weights · Details by part

Requirements · Research details and dates

On this page

Overview

R3M (facebookresearch/r3m, arXiv:2203.12601) is a frozen visual representation backbone pre-trained on Ego4D human video clips and natural-language annotations using time-contrastive learning, video-language alignment, and L1 sparsity regularization. The GitHub repository is archived (archived: true) at pinned commit b2334e726887fa0206962d7984c69c5fb09cceab (last commit August 2022, last push March 2023). [1] [3] [5]

Model card summary

Architecture

ResNet visual encoder (ResNet-18, ResNet-34, or ResNet-50) pre-trained on Ego4D video clips with time-contrastive learning (TCN), video-language alignment, and L1/L2 sparsity penalties; load_r3m() wraps the encoder in torch.nn.DataParallel and removes lang_enc/lang_rew weights for visual-only downstream use.

[1] [4] [5]

Base model

ResNet-18, ResNet-34, and ResNet-50 visual backbones (codebase adapted in part from DrQV2)

[1] [4]

Parameter count

Not explicitly stated as a scalar in README or arXiv:2203.12601 abstract; released in ResNet-18, ResNet-34, and ResNet-50 backbone variants with language head removed at load time

Exact parameter count not printed in README; Google Drive .pt payloads were not downloaded.

[1] [4] · Editorial assessment

Input modalities
rgb

224x224 RGB frames at inference; pre-training also uses Ego4D natural-language clip annotations, but load_r3m() strips the language head.

[1] [4]

Output modalities
video-frames

R3M is a frozen visual representation encoder outputting compact visual embeddings rather than direct robot actions; mapped to video-frames as the closest controlled-vocabulary visual modality.

[1] [4] [5] · Editorial assessment

Action space

Visual representation backbone only (no direct robot action output); produces compact visual feature embeddings used as frozen inputs to downstream behavior cloning or RL policies.

[1] [5]

Trained / supported embodiments
  • Embodiment-agnostic visual representation backbone (upstream paper evaluated on 12 simulated manipulation tasks and a real Franka Emika Panda arm)

[5]

Training datasets
  • Ego4D human video dataset parsed into 224x224 video clip frames paired with natural language annotations (manifest.csv)

[1] [5]

Interfaces
  • Python API (r3m.load_r3m)
  • Hydra CLI (train_representation.py)

[1] [4]

Checkpoints

CheckpointFormatSizeAccessPinned revision
R3M ResNet-50 (r3m_50)pytorchNot statedopenmain / b2334e726887
R3M ResNet-34 (r3m_34)pytorchNot statedopenmain / b2334e726887
R3M ResNet-18 (r3m_18)pytorchNot statedopenmain / b2334e726887

Three primary Google Drive PyTorch checkpoints defined in r3m/__init__.py (load_r3m); payloads were not downloaded so exact file sizes are unknown. Four additional ablation checkpoints exist in load_r3m_reproduce().

[1] [4]

Is this for you?

Use cases
  • Frozen visual feature extraction for simulated and real-world robotic manipulation behavior cloning
  • Reproducing baseline pre-trained visual representation comparisons against newer encoders

[1] [5]

Known limitations
  • GitHub repository is archived (read-only) with last push in March 2023 and no ongoing dependency maintenance.
  • Pre-trained checkpoints are downloaded via gdown from individual Google Drive file IDs rather than a versioned model hub, and external weight payloads were not downloaded or verified by OSR.
  • r3m/__init__.py strips the language encoder and reward head at load time (load_r3m) and contains an undefined-variable bug ('modelif' instead of 'modelid' in elif branches, raising NameError if evaluated) in load_r3m_reproduce() for r3m_nol1/r3m_nolang.
  • Training from scratch requires obtaining and pre-parsing the external Ego4D dataset into 224x224 frame directories and manifest.csv.

[1] [3] [4]

Safety and limitations
  • Archived repository (read-only since 2023) with pinned legacy conda dependencies (r3m_base.yaml) and gdown-based Google Drive checkpoint fetches that may break or rate-limit.
  • R3M is a frozen visual feature encoder only and outputs no robot control commands; downstream policy safety and hardware limits must be engineered separately.
  • External Google Drive checkpoints carry no separate verified license file inside this review; Ego4D pre-training data carries its own separate dataset license.
  • Open Source Robotics has not installed R3M, downloaded checkpoints, run inference, or reproduced manipulation benchmarks.

[1] [2] [3] [4]

Compute & fine-tuning

Inference compute
  • Documented hardware: PyTorch CUDA GPU or CPU (r3m/__init__.py selects cuda if available else cpu; exact VRAM and latency not stated in README)
  • Edge / consumer capable: No

load_r3m() supports CUDA or CPU in PyTorch; no minimum VRAM, latency, or edge benchmark is documented.

[4] · Editorial assessment

Fine-tuning
  • Supported: Yes
  • Methods: pre-training from scratch via train_representation.py (Hydra + Ego4D); downstream behavior cloning on frozen representation (eval branch)
  • Recipe: Upstream fine-tuning documentation

[1]

Host computer

PyTorch CPU or CUDA GPU host for inference (r3m/__init__.py falls back to CPU if CUDA is unavailable); multi-hour GPU compute and parsed Ego4D storage required for representation pre-training.

Inferred from r3m/__init__.py device selection and README training instructions.

[1] [4] · Editorial assessment

Operating system
  • Linux

Conda YAML environment and standard robotics research stack target Linux.

[1] · Editorial assessment

Dependencies
  • PyTorch
  • Hydra and OmegaConf
  • gdown (for Google Drive checkpoint downloads)
  • Conda environment (r3m_base.yaml)

[1] [4]

Skills
  • Python, PyTorch, and Conda
  • Hydra / OmegaConf configuration
  • Robotic imitation learning / behavior cloning

Inferred from README and r3m/__init__.py imports.

[1] [4] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1]

  • Create a conda environment from r3m/r3m_base.yaml (or use an existing PyTorch conda environment).
  • Run pip install -e . from the repository root and verify import r3m.
  • Call r3m.load_r3m('resnet50') (or 'resnet34', 'resnet18') to fetch the Google Drive model.pt and config.yaml via gdown into ~/.r3m/.
  • Use the frozen encoder in eval() mode as the visual frontend for downstream behavior cloning (see the separate eval branch).

[1] [4]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
12 Simulated Robot Manipulation Tasks (arXiv:2203.12601)Improves task success by >20% compared to training from scratch and >10% compared to CLIP and MoCo.Upstream-reported paper abstract summary; OSR has not reproduced behavior-cloning evaluations.
Real-World Franka Emika Panda Manipulation in Cluttered Apartment (arXiv:2203.12601)Enables learning a range of manipulation tasks given 20 demonstrations per task.Upstream-reported hardware evaluation in paper abstract; not independently verified by OSR.

[5]

Project notes

Architecture, checkpoints, and code caveats

R3M provides three primary ResNet backbones—resnet50, resnet34, and resnet18—loaded via r3m.load_r3m(), which downloads model.pt and config.yaml from hardcoded Google Drive IDs using gdown. At load time, r3m/__init__.py explicitly strips the language encoder and reward head (lang_enc, lang_rew) and wraps the visual encoder in torch.nn.DataParallel so downstream policies consume compact visual embeddings from 224×224 RGB frames. [1] [4]

Because the repository is archived and fetches checkpoints via gdown from Google Drive, users face legacy dependency pinning (r3m_base.yaml) and the possible risk of Google Drive rate limiting or link rot; current Google Drive download reliability and compatibility with modern PyTorch/CUDA stacks have not been verified in this review. Separately from the primary load_r3m() path, load_r3m_reproduce() defines four reproduction/ablation IDs (r3m, r3m_noaug, r3m_nol1, and r3m_nolang): r3m and r3m_noaug are checked first, whereas execution reaching the r3m_nol1 or r3m_nolang branches hits an undefined-variable bug (modelif instead of modelid) that raises a NameError (download availability for all Google Drive IDs remains unverified). Pre-training from scratch requires obtaining and parsing the external Ego4D dataset into per-clip 224×224 frame directories and a manifest.csv index. [1] [3] [4]

Upstream-reported evaluation

In arXiv:2203.12601, the authors report that using R3M as a frozen visual encoder across 12 simulated robot manipulation tasks improves task success by over 20% compared to training from scratch and by over 10% compared to CLIP and MoCo, and enables a real Franka Emika Panda arm to learn manipulation tasks in a cluttered apartment from 20 demonstrations. All reported results are upstream-reported; Open Source Robotics has not installed R3M, downloaded the Google Drive checkpoints, run training, fine-tuning, or inference, or reproduced any benchmarks. [5]

What we checked

Source researched only; no installation, training, fine-tuning, inference, benchmark, or physical hardware test performed. No affiliate links or reused media.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Mar 21, 2023 [3]
Latest release
Not established
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
archived commit snapshot; no GitHub release / b2334e726887fa0206962d7984c69c5fb09cceab · software repository

Separate license matrix

Include the MIT-licensed R3M visual representation codebase as a partially open archived robotics baseline; code is MIT, while external Google Drive checkpoints lack a verified standalone license and Ego4D pre-training data is external.

LayerAvailability / license
Codeavailable · MIT
Permissive MIT License in root LICENSE; repository is archived. [2] [3]
Model weightsavailable · License not fully assessed
Three ResNet checkpoints are fetched via gdown from Google Drive URLs in r3m/__init__.py; payloads were not downloaded and no separate artifact-specific weight license file was verified. [1] [4]
Training datasetsnot found · License not fully assessed
Pre-trained on the external Ego4D dataset, which is not redistributed in this repository and carries its own separate access agreement. [1] [5]
Documentationavailable · MIT
README and example scripts in archived repository. [1] [2]

Licensing and dataset separation

The repository code and documentation carry the permissive MIT License (LICENSE). However, the pre-trained checkpoints are distributed externally via Google Drive without a separate verified license file inside this review, and the underlying Ego4D pre-training dataset is governed by its own separate dataset license and access terms. [1] [2] [4] [5]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile for R3M (archived MIT repo, 3 ResNet checkpoints).