Open Source RoboticsRobots, software & learning

CoTracker3

Robotics model

Transformer-based dense and arbitrary-pixel 2D video point tracker from Meta AI and Oxford VGG, offering online (sliding-window) and offline (single-window) checkpoints trained with real-video pseudo-labelling. [1] [5]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Robotics and computer vision researchers building point-tracking perception, visual servoing, or SfM pipelines (such as VGGSfM); Academic and non-commercial teams evaluating dense 2D point trajectories on manipulation or dynamic-scene videos [1] [2]
Parameters
Not explicitly stated in README or abstract (checkpoint files are ~101.7–101.9 MB / 0.102 GB each in PyTorch .pth… [1] [4] [5]
Embodiments
Embodiment-agnostic 2D RGB video perception module (evaluated upstream on TAP-Vid Kinetics, DAVIS, RoboTAP, and… [1] [5]
Source rights
Named licenses are documented by part. Includes noncommercial terms. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware

Requirements · Research details and dates

On this page

Overview

CoTracker3 (facebookresearch/co-tracker, arXiv:2410.11831) is a transformer-based point tracker from Meta AI Research and the University of Oxford VGG that jointly tracks arbitrary user-selected pixels or quasi-dense grids across RGB videos, including through occlusions. At pinned commit 82e02e8029753ad4ef13cf06be7f4fc5facdda4d (dated January 21, 2025, announcing the synthetic Kubric training set), the repository provides PyTorch Hub entry points, online and offline inference demos, TAP-Vid evaluation scripts, and both synthetic-pretraining and real-video pseudo-labelling pipelines. [1] [3] [5]

Model card summary

Architecture

Lightweight transformer-based point tracker with virtual tracks and correlation-based iterative updates, released in an offline single-window variant and an online sliding-window variant trained first on synthetic Kubric data and then scaled via semi-supervised pseudo-labelling of real videos.

[1] [5]

Parameter count

Not explicitly stated in README or abstract (checkpoint files are ~101.7–101.9 MB / 0.102 GB each in PyTorch .pth format)

Verified exact Hugging Face file sizes: scaled_offline.pth (101,890,938 bytes), scaled_online.pth (101,695,610 bytes), baseline_offline.pth (101,889,786 bytes), baseline_online.pth (101,694,458 bytes).

[1] [4] [5]

Input modalities
rgb

RGB video tensor (B, T, C, H, W) plus optional user-specified 2D query points or regular grid_size.

[1]

Output modalities
waypoints

Outputs 2D image-plane point trajectories pred_tracks (B, T, N, 2) and per-frame visibility predictions pred_visibility (B, T, N, 1) rather than robot control commands.

[1]

Action space

Perception-only 2D point tracker (no robot action output); produces 2D pixel coordinate trajectories (B, T, N, 2) and visibility flags (B, T, N, 1) across video frames.

[1]

Trained / supported embodiments
  • Embodiment-agnostic 2D RGB video perception module (evaluated upstream on TAP-Vid Kinetics, DAVIS, RoboTAP, and RGB-Stacking)

[1] [5]

Training datasets
  • Kubric MOVI-f synthetic sequences (6,000 high-resolution 512x512, 120-frame sequences published as facebook/CoTracker3_Kubric under Apache-2.0)
  • Unlabelled real videos pseudo-labelled via off-the-shelf teacher trackers for scaled checkpoints (real video corpus not redistributed)

[1] [5] [6]

Interfaces
  • PyTorch Hub API (cotracker3_offline, cotracker3_online)
  • Python CLI demos (demo.py, online_demo.py)
  • Gradio web app (gradio_demo/app.py) and Jupyter/Colab notebook
  • Hydra evaluation CLI and PyTorch Lightning training scripts

[1]

Checkpoints

All four CoTracker3 .pth files reside in the ungated facebook/cotracker3 Hugging Face repo (CC-BY-NC-4.0) and can also be loaded via torch.hub.load('facebookresearch/co-tracker', 'cotracker3_offline' | 'cotracker3_online').

[1] [4]

Is this for you?

Use cases
  • Tracking arbitrary 2D pixels or dense grids jointly across RGB videos (offline single-window or online sliding-window streaming)
  • Generating 2D point correspondences and visibility estimates for downstream SfM (e.g., VGGSfM) and robot manipulation perception (e.g., RoboTAP)
  • Fine-tuning on custom real video corpora using the provided pseudo-labelling training pipeline

[1] [5]

Known limitations
  • Repository code and all four CoTracker3 checkpoints are licensed under CC-BY-NC-4.0 (Non-Commercial), prohibiting commercial robotics deployment without separate licensing.
  • Outputs 2D image-plane pixel trajectories and visibility flags only; does not directly output 3D coordinates, camera poses, or robot control actions.
  • Training from scratch on Kubric is documented with 32 GPUs (4 nodes x 8 GPUs) and requires generating or downloading Kubric annotations plus collecting real videos for pseudo-label fine-tuning.
  • Open Source Robotics has not installed CoTracker3, downloaded checkpoints, run inference or fine-tuning, or reproduced TAP-Vid/RoboTAP benchmarks.

[1] [2] [4]

Safety and limitations
  • CC-BY-NC-4.0 license on both the codebase and Hugging Face checkpoints restricts use to non-commercial purposes; commercial teams cannot deploy the released weights without separate rights.
  • Perception-only 2D point tracker without metric 3D guarantees or robot safety bounds; occluded or rapidly deforming objects may lose track.
  • No independent runtime, VRAM, or TAP-Vid/RoboTAP accuracy verification was performed by Open Source Robotics.

[1] [2] [4]

Compute & fine-tuning

Inference compute
  • Documented hardware: CUDA GPU strongly recommended (small tasks can run on CPU; online sliding-window mode reduces memory for longer videos)
  • Precision: fp32 (mixed precision supported in training)
  • Edge / consumer capable: No

Exact minimum VRAM and inference latency are not specified in the README.

[1]

Fine-tuning
  • Supported: Yes
  • Methods: full training on Kubric (train_on_kubric.py); semi-supervised pseudo-label fine-tuning on custom real videos (train_on_real_data.py)
  • Recipe: Upstream fine-tuning documentation

Upstream baseline training used 32 GPUs (4 nodes x 8 GPUs) for 50,000 steps; pseudo-label fine-tuning starts from baseline_online.pth or baseline_offline.pth for 15,000 steps.

[1]

Host computer

CUDA GPU strongly recommended for inference (CPU usable for small tasks; ~0.102 GB checkpoint file); 32 GPUs (4 nodes x 8 GPUs) documented for full Kubric training.

[1] [4]

Operating system
  • Linux

Standard PyTorch CUDA and multi-node training workflow targets Linux.

[1] · Editorial assessment

Dependencies
  • PyTorch and TorchVision (CUDA strongly recommended)
  • imageio[ffmpeg], matplotlib, flow_vis, tqdm, tensorboard
  • Evaluation/training extras: hydra-core==1.1.0, mediapy, pytorch_lightning==1.6.0, opencv-python

[1]

Skills
  • Python and PyTorch
  • Video I/O and 2D point-tracking evaluation (Hydra, PyTorch Lightning for training)

Inferred from Quick Start, evaluation, and training instructions.

[1] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1] · Model card / documentation [1]

  • Install PyTorch and TorchVision (with CUDA support recommended) plus imageio[ffmpeg] for quick-start loading.
  • Load directly via torch.hub.load('facebookresearch/co-tracker', 'cotracker3_offline') or 'cotracker3_online', or clone the repo and run pip install -e .
  • For local demos, evaluation, or training, download the .pth checkpoints from facebook/cotracker3 into checkpoints/ and run demo.py or online_demo.py.

[1]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
TAP-Vid Kinetics (delta_avg^vis)CoTracker3 online: 68.3; CoTracker3 offline: 67.8 (vs CoTracker2: 61.8, CoTracker2.1: 63.0).Upstream-reported in README Evaluation table (joint point evaluation); OSR has not run evaluation or reproduced TAP-Vid numbers.
TAP-Vid DAVIS (delta_avg^vis)CoTracker3 offline: 76.9; CoTracker3 online: 76.7 (vs CoTracker2: 74.6, CoTracker2.1: 76.1).Upstream-reported in README Evaluation table; OSR has not reproduced this benchmark.
RoboTAP (delta_avg^vis)CoTracker3 online: 78.8; CoTracker3 offline: 78.0 (vs CoTracker2: 69.6, CoTracker2.1: 70.6).Upstream-reported in README Evaluation table on robotic manipulation tracking benchmark; OSR has not reproduced this benchmark.
RGB-Stacking (delta_avg^vis)CoTracker3 offline: 85.0; CoTracker3 online: 82.7 (vs CoTracker2: 73.4, CoTracker2.1: 79.6).Upstream-reported in README Evaluation table; OSR has not reproduced this benchmark.

[1]

Project notes

Architecture, checkpoints, and workflows

CoTracker3 simplifies earlier CoTracker architectures and introduces a semi-supervised training recipe that generates pseudo-labels on unlabelled real videos using off-the-shelf teacher models, reducing the synthetic-to-real gap while using 1,000× less data than prior top-performing trackers. The model takes an RGB video tensor (B, T, C, H, W) with either manual 2D query points or a regular grid and outputs 2D pixel trajectories (B, T, N, 2) alongside per-frame visibility flags (B, T, N, 1). It is a perception-only 2D tracker and does not directly output 3D poses or robot control commands. [1] [5]

Four ungated PyTorch .pth checkpoints (approximately 101.7–101.9 MB / 0.102 GB each on disk; checkpoint file size does not establish runtime GPU memory requirements) are hosted in facebook/cotracker3 on Hugging Face (revision bf55ea50d4390e1820a267f131cd6587240fb2c5): scaled_online.pth (sliding-window online mode) and scaled_offline.pth (single-window offline mode) trained on Kubric plus pseudo-labelled real videos, alongside Kubric-only baseline_online.pth and baseline_offline.pth. Both scaled variants can be loaded directly via torch.hub.load("facebookresearch/co-tracker", "cotracker3_online" | "cotracker3_offline"). Online sliding-window processing lowers memory consumption for longer streams, though exact minimum VRAM and streaming frame rates are not stated in the README or measured by this review. Full training scripts (train_on_kubric.py, documented on 32 GPUs across 4 nodes, and train_on_real_data.py for pseudo-label fine-tuning) are included. [1] [4]

Upstream-reported benchmarks

The README reports $\delta_{\text{avg}}^{\text{vis}}$ accuracy across four benchmarks (joint point evaluation): on TAP-Vid Kinetics, CoTracker3 online reaches 68.3 and offline 67.8 (vs. 61.8 for CoTracker2); on TAP-Vid DAVIS, offline reaches 76.9 and online 76.7; on the robotic manipulation benchmark RoboTAP, online reaches 78.8 and offline 78.0 (up from 69.6 for CoTracker2); and on RGB-Stacking, offline reaches 85.0 and online 82.7. All scores are upstream-reported; Open Source Robotics has not installed CoTracker3, downloaded weights, run training, fine-tuning, or inference, or reproduced any benchmark numbers. [1]

What we checked

Source researched only; no installation, training, fine-tuning, inference, benchmark, or physical hardware test performed. No affiliate links or reused media.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Jan 21, 2025 [3]
Latest release
Not established
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
commit snapshot; no GitHub release / 82e02e8029753ad4ef13cf06be7f4fc5facdda4d · software repository

Separate license matrix

Include CoTracker3 as a widely used robotics and 3D-vision point tracker with public checkpoints, training code, and synthetic Kubric dataset, classified as source-available because the codebase and weights are licensed under CC-BY-NC-4.0 (non-commercial).

LayerAvailability / license
Codeavailable · CC-BY-NC-4.0 (with portions under MIT and Apache-2.0)
Root LICENSE.md is CC-BY-NC-4.0 (non-commercial); README notes Particle Video Revisited portions are MIT and TAP-Vid/LocoTrack portions are Apache-2.0. [1] [2] [3]
Model weightsavailable · CC-BY-NC-4.0
Ungated Hugging Face repo tagged license:cc-by-nc-4.0 containing scaled and baseline online/offline .pth checkpoints. [2] [4]
Training datasetsavailable · Apache-2.0 (CoTracker3_Kubric synthetic dataset; real pseudo-labelled videos not redistributed)
6,000 synthetic Kubric sequences are published ungated under Apache-2.0 on Hugging Face; users must supply their own real video datasets for pseudo-label fine-tuning. [1] [6]
Documentationavailable · CC-BY-NC-4.0
README, Sphinx docs directory, Colab notebook, and Gradio demo included. [1] [2]

Licensing and dataset availability

The repository LICENSE.md and the facebook/cotracker3 Hugging Face checkpoints are licensed under CC-BY-NC-4.0. Under Section 2(a)(1), users may reproduce, share, and produce adapted material worldwide and royalty-free for non-commercial purposes only; when sharing the material (including modified versions), Section 3(a)(1) requires retaining supplied attribution and copyright/license/warranty notices, indicating modifications, including the license text or a link to it, and retaining a URI to the material where reasonably practicable. Commercial robotics deployment is not permitted under these terms without separate rights. Portions of the repository adapted from Particle Video Revisited are MIT-licensed and portions from TAP-Vid and LocoTrack are Apache-2.0. Notably, the accompanying 6,000-sequence synthetic dataset (facebook/CoTracker3_Kubric, 512×512, 120 frames) is published ungated under Apache-2.0, whereas the real videos used for pseudo-labelling are not redistributed and must be supplied by users who wish to fine-tune. [1] [2] [4] [6]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile for CoTracker3 (4 verified PyTorch checkpoints, Apache-2.0 Kubric dataset vs CC-BY-NC-4.0 code and weights).