CoTracker3
Robotics modelTransformer-based dense and arbitrary-pixel 2D video point tracker from Meta AI and Oxford VGG, offering online (sliding-window) and offline (single-window) checkpoints trained with real-video pseudo-labelling. [1] [5]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Robotics and computer vision researchers building point-tracking perception, visual servoing, or SfM pipelines (such as VGGSfM); Academic and non-commercial teams evaluating dense 2D point trajectories on manipulation or dynamic-scene videos [1] [2]
- Parameters
- Not explicitly stated in README or abstract (checkpoint files are ~101.7–101.9 MB / 0.102 GB each in PyTorch .pth… [1] [4] [5]
- Embodiments
- Embodiment-agnostic 2D RGB video perception module (evaluated upstream on TAP-Vid Kinetics, DAVIS, RoboTAP, and… [1] [5]
- Source rights
- Named licenses are documented by part. Includes noncommercial terms. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- License
- CC-BY-NC-4.0 (with portions under MIT and Apache-2.0) (noncommercial), CC-BY-NC-4.0 (noncommercial), Apache-2.0 (CoTracker3_Kubric synthetic dataset; real pseudo-labelled videos not redistributed) · Details by part
Overview
CoTracker3 (facebookresearch/co-tracker, arXiv:2410.11831) is a transformer-based point tracker from Meta AI Research and the University of Oxford VGG that jointly tracks arbitrary user-selected pixels or quasi-dense grids across RGB videos, including through occlusions. At pinned commit 82e02e8029753ad4ef13cf06be7f4fc5facdda4d (dated January 21, 2025, announcing the synthetic Kubric training set), the repository provides PyTorch Hub entry points, online and offline inference demos, TAP-Vid evaluation scripts, and both synthetic-pretraining and real-video pseudo-labelling pipelines. [1] [3] [5]
Model card summary
- Architecture
Lightweight transformer-based point tracker with virtual tracks and correlation-based iterative updates, released in an offline single-window variant and an online sliding-window variant trained first on synthetic Kubric data and then scaled via semi-supervised pseudo-labelling of real videos.
- Parameter count
Not explicitly stated in README or abstract (checkpoint files are ~101.7–101.9 MB / 0.102 GB each in PyTorch .pth format)
Verified exact Hugging Face file sizes: scaled_offline.pth (101,890,938 bytes), scaled_online.pth (101,695,610 bytes), baseline_offline.pth (101,889,786 bytes), baseline_online.pth (101,694,458 bytes).
- Input modalities
- rgb
RGB video tensor (B, T, C, H, W) plus optional user-specified 2D query points or regular grid_size.
- Output modalities
- waypoints
Outputs 2D image-plane point trajectories pred_tracks (B, T, N, 2) and per-frame visibility predictions pred_visibility (B, T, N, 1) rather than robot control commands.
- Action space
Perception-only 2D point tracker (no robot action output); produces 2D pixel coordinate trajectories (B, T, N, 2) and visibility flags (B, T, N, 1) across video frames.
- Trained / supported embodiments
- Embodiment-agnostic 2D RGB video perception module (evaluated upstream on TAP-Vid Kinetics, DAVIS, RoboTAP, and RGB-Stacking)
- Training datasets
- Kubric MOVI-f synthetic sequences (6,000 high-resolution 512x512, 120-frame sequences published as facebook/CoTracker3_Kubric under Apache-2.0)
- Unlabelled real videos pseudo-labelled via off-the-shelf teacher trackers for scaled checkpoints (real video corpus not redistributed)
- Interfaces
- PyTorch Hub API (cotracker3_offline, cotracker3_online)
- Python CLI demos (demo.py, online_demo.py)
- Gradio web app (gradio_demo/app.py) and Jupyter/Colab notebook
- Hydra evaluation CLI and PyTorch Lightning training scripts
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| facebook/cotracker3 — scaled_online.pth (online sliding-window, Kubric + real pseudo-labels) | pytorch | 0.102 GB | open | bf55ea50d439 |
| facebook/cotracker3 — scaled_offline.pth (offline single-window, Kubric + real pseudo-labels) | pytorch | 0.102 GB | open | bf55ea50d439 |
| facebook/cotracker3 — baseline_online.pth (online sliding-window, Kubric only) | pytorch | 0.102 GB | open | bf55ea50d439 |
| facebook/cotracker3 — baseline_offline.pth (offline single-window, Kubric only) | pytorch | 0.102 GB | open | bf55ea50d439 |
All four CoTracker3 .pth files reside in the ungated facebook/cotracker3 Hugging Face repo (CC-BY-NC-4.0) and can also be loaded via torch.hub.load('facebookresearch/co-tracker', 'cotracker3_offline' | 'cotracker3_online').
Is this for you?
- Use cases
- Tracking arbitrary 2D pixels or dense grids jointly across RGB videos (offline single-window or online sliding-window streaming)
- Generating 2D point correspondences and visibility estimates for downstream SfM (e.g., VGGSfM) and robot manipulation perception (e.g., RoboTAP)
- Fine-tuning on custom real video corpora using the provided pseudo-labelling training pipeline
- Known limitations
- Repository code and all four CoTracker3 checkpoints are licensed under CC-BY-NC-4.0 (Non-Commercial), prohibiting commercial robotics deployment without separate licensing.
- Outputs 2D image-plane pixel trajectories and visibility flags only; does not directly output 3D coordinates, camera poses, or robot control actions.
- Training from scratch on Kubric is documented with 32 GPUs (4 nodes x 8 GPUs) and requires generating or downloading Kubric annotations plus collecting real videos for pseudo-label fine-tuning.
- Open Source Robotics has not installed CoTracker3, downloaded checkpoints, run inference or fine-tuning, or reproduced TAP-Vid/RoboTAP benchmarks.
- Safety and limitations
- CC-BY-NC-4.0 license on both the codebase and Hugging Face checkpoints restricts use to non-commercial purposes; commercial teams cannot deploy the released weights without separate rights.
- Perception-only 2D point tracker without metric 3D guarantees or robot safety bounds; occluded or rapidly deforming objects may lose track.
- No independent runtime, VRAM, or TAP-Vid/RoboTAP accuracy verification was performed by Open Source Robotics.
Compute & fine-tuning
- Inference compute
- Documented hardware: CUDA GPU strongly recommended (small tasks can run on CPU; online sliding-window mode reduces memory for longer videos)
- Precision:
fp32 (mixed precision supported in training) - Edge / consumer capable: No
Exact minimum VRAM and inference latency are not specified in the README.
- Fine-tuning
- Supported: Yes
- Methods: full training on Kubric (train_on_kubric.py); semi-supervised pseudo-label fine-tuning on custom real videos (train_on_real_data.py)
- Recipe: Upstream fine-tuning documentation
Upstream baseline training used 32 GPUs (4 nodes x 8 GPUs) for 50,000 steps; pseudo-label fine-tuning starts from baseline_online.pth or baseline_offline.pth for 15,000 steps.
- Host computer
CUDA GPU strongly recommended for inference (CPU usable for small tasks; ~0.102 GB checkpoint file); 32 GPUs (4 nodes x 8 GPUs) documented for full Kubric training.
- Operating system
- Linux
Standard PyTorch CUDA and multi-node training workflow targets Linux.
[1] · Editorial assessment
- Dependencies
- PyTorch and TorchVision (CUDA strongly recommended)
- imageio[ffmpeg], matplotlib, flow_vis, tqdm, tensorboard
- Evaluation/training extras: hydra-core==1.1.0, mediapy, pytorch_lightning==1.6.0, opencv-python
- Skills
- Python and PyTorch
- Video I/O and 2D point-tracking evaluation (Hydra, PyTorch Lightning for training)
Inferred from Quick Start, evaluation, and training instructions.
[1] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [1]
- Install PyTorch and TorchVision (with CUDA support recommended) plus imageio[ffmpeg] for quick-start loading.
- Load directly via torch.hub.load('facebookresearch/co-tracker', 'cotracker3_offline') or 'cotracker3_online', or clone the repo and run pip install -e .
- For local demos, evaluation, or training, download the .pth checkpoints from facebook/cotracker3 into checkpoints/ and run demo.py or online_demo.py.
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| TAP-Vid Kinetics (delta_avg^vis) | CoTracker3 online: 68.3; CoTracker3 offline: 67.8 (vs CoTracker2: 61.8, CoTracker2.1: 63.0). | Upstream-reported in README Evaluation table (joint point evaluation); OSR has not run evaluation or reproduced TAP-Vid numbers. |
| TAP-Vid DAVIS (delta_avg^vis) | CoTracker3 offline: 76.9; CoTracker3 online: 76.7 (vs CoTracker2: 74.6, CoTracker2.1: 76.1). | Upstream-reported in README Evaluation table; OSR has not reproduced this benchmark. |
| RoboTAP (delta_avg^vis) | CoTracker3 online: 78.8; CoTracker3 offline: 78.0 (vs CoTracker2: 69.6, CoTracker2.1: 70.6). | Upstream-reported in README Evaluation table on robotic manipulation tracking benchmark; OSR has not reproduced this benchmark. |
| RGB-Stacking (delta_avg^vis) | CoTracker3 offline: 85.0; CoTracker3 online: 82.7 (vs CoTracker2: 73.4, CoTracker2.1: 79.6). | Upstream-reported in README Evaluation table; OSR has not reproduced this benchmark. |
Project notes
Architecture, checkpoints, and workflows
CoTracker3 simplifies earlier CoTracker architectures and introduces a semi-supervised training recipe that generates pseudo-labels on unlabelled real videos using off-the-shelf teacher models, reducing the synthetic-to-real gap while using 1,000× less data than prior top-performing trackers. The model takes an RGB video tensor (B, T, C, H, W) with either manual 2D query points or a regular grid and outputs 2D pixel trajectories (B, T, N, 2) alongside per-frame visibility flags (B, T, N, 1). It is a perception-only 2D tracker and does not directly output 3D poses or robot control commands. [1] [5]
Four ungated PyTorch .pth checkpoints (approximately 101.7–101.9 MB / 0.102 GB each on disk; checkpoint file size does not establish runtime GPU memory requirements) are hosted in facebook/cotracker3 on Hugging Face (revision bf55ea50d4390e1820a267f131cd6587240fb2c5): scaled_online.pth (sliding-window online mode) and scaled_offline.pth (single-window offline mode) trained on Kubric plus pseudo-labelled real videos, alongside Kubric-only baseline_online.pth and baseline_offline.pth. Both scaled variants can be loaded directly via torch.hub.load("facebookresearch/co-tracker", "cotracker3_online" | "cotracker3_offline"). Online sliding-window processing lowers memory consumption for longer streams, though exact minimum VRAM and streaming frame rates are not stated in the README or measured by this review. Full training scripts (train_on_kubric.py, documented on 32 GPUs across 4 nodes, and train_on_real_data.py for pseudo-label fine-tuning) are included. [1] [4]
Upstream-reported benchmarks
The README reports $\delta_{\text{avg}}^{\text{vis}}$ accuracy across four benchmarks (joint point evaluation): on TAP-Vid Kinetics, CoTracker3 online reaches 68.3 and offline 67.8 (vs. 61.8 for CoTracker2); on TAP-Vid DAVIS, offline reaches 76.9 and online 76.7; on the robotic manipulation benchmark RoboTAP, online reaches 78.8 and offline 78.0 (up from 69.6 for CoTracker2); and on RGB-Stacking, offline reaches 85.0 and online 82.7. All scores are upstream-reported; Open Source Robotics has not installed CoTracker3, downloaded weights, run training, fine-tuning, or inference, or reproduced any benchmark numbers. [1]
What we checked
Source researched only; no installation, training, fine-tuning, inference, benchmark, or physical hardware test performed. No affiliate links or reused media.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Upstream last updated
- Jan 21, 2025 [3]
- Latest release
- Not established
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- commit snapshot; no GitHub release / 82e02e8029753ad4ef13cf06be7f4fc5facdda4d · software repository
Separate license matrix
Include CoTracker3 as a widely used robotics and 3D-vision point tracker with public checkpoints, training code, and synthetic Kubric dataset, classified as source-available because the codebase and weights are licensed under CC-BY-NC-4.0 (non-commercial).
| Layer | Availability / license |
|---|---|
| Code | available · CC-BY-NC-4.0 (with portions under MIT and Apache-2.0) Root LICENSE.md is CC-BY-NC-4.0 (non-commercial); README notes Particle Video Revisited portions are MIT and TAP-Vid/LocoTrack portions are Apache-2.0. [1] [2] [3] |
| Model weights | available · CC-BY-NC-4.0 Ungated Hugging Face repo tagged license:cc-by-nc-4.0 containing scaled and baseline online/offline .pth checkpoints. [2] [4] |
| Training datasets | available · Apache-2.0 (CoTracker3_Kubric synthetic dataset; real pseudo-labelled videos not redistributed) 6,000 synthetic Kubric sequences are published ungated under Apache-2.0 on Hugging Face; users must supply their own real video datasets for pseudo-label fine-tuning. [1] [6] |
| Documentation | available · CC-BY-NC-4.0 README, Sphinx docs directory, Colab notebook, and Gradio demo included. [1] [2] |
Licensing and dataset availability
The repository LICENSE.md and the facebook/cotracker3 Hugging Face checkpoints are licensed under CC-BY-NC-4.0. Under Section 2(a)(1), users may reproduce, share, and produce adapted material worldwide and royalty-free for non-commercial purposes only; when sharing the material (including modified versions), Section 3(a)(1) requires retaining supplied attribution and copyright/license/warranty notices, indicating modifications, including the license text or a link to it, and retaining a URI to the material where reasonably practicable. Commercial robotics deployment is not permitted under these terms without separate rights. Portions of the repository adapted from Particle Video Revisited are MIT-licensed and portions from TAP-Vid and LocoTrack are Apache-2.0. Notably, the accompanying 6,000-sequence synthetic dataset (facebook/CoTracker3_Kubric, 512×512, 120 frames) is published ungated under Apache-2.0, whereas the real videos used for pseudo-labelling are not redistributed and must be supplied by users who wish to fine-tune. [1] [2] [4] [6]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 CoTracker3 README
PyTorch Hub quick start, online/offline tracking APIs, four checkpoint links, TAP-Vid/RoboTAP/RGB-Stacking evaluation table, Kubric and pseudo-label training recipes, and license summary.Revision, date & where cited
facebookresearch/co-tracker contributors · github.com · revision
82e02e8· Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 5 more - S002 CoTracker3 LICENSE.md (CC-BY-NC-4.0)
Full text of Creative Commons Attribution-NonCommercial 4.0 International Public License.Revision, date & where cited
Meta AI Research / Creative Commons · github.com · revision
82e02e8· Accessed Nov 21, 2026
Used for: Is this for you?, Separate license matrix - S003 GitHub repository metadata and pinned commit
Last commit 2025-01-21; no GitHub releases; not archived.Revision, date & where cited
facebookresearch/co-tracker / GitHub · github.com · revision
82e02e8· main / · Accessed Nov 21, 2026
Used for: Overview, What we checked, Separate license matrix - S004 Hugging Face facebook/cotracker3 checkpoint metadata
Verified four ungated .pth checkpoints (scaled_online, scaled_offline, baseline_online, baseline_offline; ~97 MB / 0.095 GB each) tagged license:cc-by-nc-4.0.Revision, date & where cited
Meta AI / Hugging Face · huggingface.co · API metadata 2026-11-21 / · Accessed Nov 21, 2026
Used for: Model card summary, Checkpoints, Is this for you?, Compute & fine-tuning and 2 more - S005 CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos (arXiv:2410.11831)
Introduces simplified online and offline CoTracker3 architectures trained with semi-supervised pseudo-labelling of real videos using 1,000x less data than prior top-performing models.Revision, date & where cited
Meta AI Research & University of Oxford VGG (arXiv) · arxiv.org · arXiv:2410.11831v1 abstract · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Architecture, checkpoints, and workflows - S006 Hugging Face facebook/CoTracker3_Kubric dataset metadata
Verified ungated dataset of 6,000 synthetic Kubric sequences tagged license:apache-2.0.Revision, date & where cited
Meta AI / Hugging Face · huggingface.co · API metadata 2026-11-21 / a23b6e6e33 · Accessed Nov 21, 2026
Used for: Model card summary, Separate license matrix
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile for CoTracker3 (4 verified PyTorch checkpoints, Apache-2.0 Kubric dataset vs CC-BY-NC-4.0 code and weights).