FoundationPose
Robotics modelResearch implementation for 6D object pose estimation and tracking from RGB-D inputs, using a CAD model or reference views. [1]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Researchers evaluating object-pose perception [1]
- Parameters
- Not stated in upstream README or arXiv:2312.08344 abstract (two separate PyTorch networks: RefineNet refiner and… [1] [10] [11]
- Embodiments
- Embodiment-agnostic perception module (RGB-D camera input [1]
- Source rights
- Some licenses are unresolved. Includes noncommercial terms. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- License
- NVIDIA Source Code License (noncommercial research/evaluation) (noncommercial) · license not fully assessed for model weights, datasets · Details by part
Overview
FoundationPose (NVlabs/FoundationPose, CVPR 2024 Highlight, arXiv:2312.08344) is a unified 6D object pose estimation and tracking model supporting both model-based (CAD mesh) and model-free (few-shot reference views via a neural implicit representation) setups without test-time fine-tuning. At pinned commit a1b694b83e633c2cb6115b9063d940a687759392, its core networks combine ResNet feature stages with Transformer encoder layers in a pose refiner (RefineNet) and multi-pair pose scorer (ScoreNetMultiPair). [1] [6] [10] [11]
Model card summary
- Architecture
Unified 6D pose estimation and tracking architecture pairing a ResNet + Transformer pose refiner (RefineNet) and multi-pair pose scorer (ScoreNetMultiPair), bridged to model-free few-shot operation via a neural implicit representation for novel view synthesis.
- Parameter count
Not stated in upstream README or arXiv:2312.08344 abstract (two separate PyTorch networks: RefineNet refiner and ScoreNetMultiPair scorer)
Weight binaries on Google Drive were not downloaded to count parameters.
- Input modalities
- rgbd
RGB-D frames plus camera intrinsics, initial object segmentation mask, and either a 3D CAD mesh or reference views.
- Action space
Perception-only model (no robot action output); produces per-frame 4x4 SE(3) object-in-camera 6D pose matrices requiring an external robot controller or grasp planner.
- Trained / supported embodiments
- Embodiment-agnostic perception module (RGB-D camera input; upstream links separate NVIDIA-ISAAC-ROS/isaac_ros_pose_estimation for ROS 2 / TensorRT C++ deployment)
- Training datasets
- FoundationPose synthetic dataset rendered from GSO and Objaverse 3D assets with domain randomization (public release excludes Stable-Diffusion texture-augmented data and weights trained on it)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| FoundationPose Refiner (2023-10-28-18-33-37) | pytorch | Not stated | open | 2023-10-28-18-33-37 |
| FoundationPose Scorer (2024-01-11-20-02-45) | pytorch | Not stated | open | 2024-01-11-20-02-45 |
Public Google Drive folder listing reachable; payloads were not downloaded, so exact file sizes are unknown. Public weights omit diffusion-augmented training data.
Is this for you?
- Use cases
- Zero-shot 6D object pose estimation from RGB-D and a CAD mesh (model-based)
- Few-shot 6D pose estimation from reference views via neural implicit rendering (model-free)
- Frame-to-frame 6D object pose tracking for robotics manipulation and AR
- Known limitations
- Repository license limits use to noncommercial research or evaluation.
- External weights and data were not downloaded or validated in this review.
- run_demo.py removes existing debug-directory contents at startup.
- The released model excludes diffusion-augmented training data and weights.
- Safety and limitations
- Repository NVIDIA Source Code License restricts use to noncommercial research or evaluation.
- Released weights exclude Stable-Diffusion texture-augmented training data due to LAION/Stable-Diffusion legal restrictions; upstream notes slight performance degradation is expected.
- run_demo.py removes existing contents of --debug_dir at startup via a shell command; always pass a disposable path without shell metacharacters.
- External Google Drive weights and demo payloads were not downloaded, executed, or benchmarked by Open Source Robotics.
Compute & fine-tuning
- Inference compute
- Documented hardware: NVIDIA CUDA GPU (PyTorch, PyTorch3D, NVDiffRast; separate Isaac ROS repo offers TensorRT C++ inference)
- Edge / consumer capable: No
Requires CUDA GPU for PyTorch3D/NVDiffRast rendering; exact minimum VRAM and latency_ms not stated in README.
- Fine-tuning
- Supported: No
Designed for zero-shot test-time application to novel objects without fine-tuning; training scripts are not part of the documented README workflow.
- Host computer
CUDA-capable NVIDIA GPU required for PyTorch, PyTorch3D, NVDiffRast, and compiled C++/CUDA extensions; exact minimum VRAM not stated in README.
- Operating system
- Linux
Docker and Conda Linux build routes documented; Windows community issue linked.
Setup & usage
Open the upstream getting-started guide [1]
- Prepare the documented CUDA environment and build extensions.
- Obtain the named scorer/refiner weights and demo data.
- Provide a mesh, RGB-D sequence, camera intrinsics and initial object mask.
- Run run_demo.py with a disposable debug directory and inspect saved pose matrices.
Editorial assessment from cited documentation; no runtime testing.
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| BOP23 Core Datasets Leaderboard (Model-based Novel Object Pose Estimation) | Ranked #1 as of March 2024 per upstream README (CVPR 2024 Highlight, arXiv:2312.08344) | Upstream-reported leaderboard standing; public weights exclude diffusion-augmented training data so upstream notes slight performance degradation is expected. OSR has not reproduced BOP evaluations. |
Project notes
Inputs, checkpoints, and setup boundaries
FoundationPose is a perception-only model: the mesh-based demo run_demo.py takes RGB-D frames, camera intrinsics, an initial object segmentation mask, and a 3D mesh, and outputs per-frame 4×4 SE(3) object-in-camera pose matrices without generating robot control commands. Model-free operation is handled separately from run_demo.py: the README documents first training a per-object Neural Object Field from reference views (object-representation preparation distinct from fine-tuning the pose networks) and then invoking the dataset-runner workflow with the reconstructed mesh and reference-view arguments, noting that build_all_conda.sh omits the BundleSDF CUDA operations required for that model-free path. The README links two external PyTorch weight directories on Google Drive—2023-10-28-18-33-37 (refiner) and 2024-01-11-20-02-45 (scorer)—plus demo data and synthetic GSO/Objaverse training data. The Google Drive folder listings were verified reachable, but binary payloads were not downloaded, so exact parameter counts and checkpoint sizes are not established here. [1] [2] [4] [8] [9]
Installation requires a CUDA-capable NVIDIA GPU with PyTorch, PyTorch3D, NVDiffRast, and compiled C++/CUDA extensions via either the documented Docker or Conda route (upstream also links a separate NVIDIA-ISAAC-ROS/isaac_ros_pose_estimation repo for ROS 2 / TensorRT C++ deployment). Note that run_demo.py deletes existing --debug_dir contents at startup via a shell command, so users should pass a disposable directory path without spaces or shell metacharacters. [1] [2] [4]
Upstream-reported benchmarks and release omissions
The README reports a #1 standing on the BOP23 core datasets leaderboard for model-based novel object pose estimation (as of March 2024). However, upstream explicitly notes that Stable-Diffusion texture-augmented training data and weights trained on it could not be released due to LAION/Stable-Diffusion legal restrictions, and that slight performance degradation is expected for the public checkpoints. All benchmark results are upstream-reported; Open Source Robotics has not installed dependencies, downloaded weights, run inference, or reproduced BOP evaluations. [1] [10]
What we checked
Source researched only; no installation, training, benchmark or physical hardware test performed. No affiliate links or reused media.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Upstream last updated
- Apr 29, 2026 [6]
- Latest release
- Not established
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- commit snapshot; no stable GitHub release / a1b694b83e633c2cb6115b9063d940a687759392 · software repository
Separate license matrix
Include the inspectable robotics pose-estimation implementation under a restricted research scope; distinguish source availability from unverified external weights and production rights.
| Layer | Availability / license |
|---|---|
| Code | available · NVIDIA Source Code License (noncommercial research/evaluation) Repository materials only; external components retain their own terms. [7] |
| Model weights | available · License not fully assessed Public Drive folder listing retrieved successfully; payloads not downloaded or validated and artifact-specific terms remain unverified. [1] [8] |
| Training datasets | available · License not fully assessed Public Drive folder listing retrieved successfully; payloads not downloaded or validated and artifact-specific terms remain unverified. [1] [9] |
| Documentation | available · NVIDIA Source Code License (noncommercial research/evaluation) Repository materials only; external components retain their own terms. [7] |
Noncommercial research license
The repository LICENSE (NVIDIA Source Code License) restricts use to noncommercial research or evaluation—not permissive commercial production deployment—and the README states that code and data are released under that license. However, this review has not verified artifact-specific license terms accompanying the external Google Drive weights or dataset payloads, and third-party dependencies retain their own separate licensing terms. [1] [7] [8] [9]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 Setup, artifacts and release limitations
Revision, date & where cited
NVlabs/FoundationPose contributors · github.com · revision
a1b694b· Accessed Oct 18, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 6 more - S002 Model-based demonstration
Revision, date & where cited
NVlabs/FoundationPose contributors · github.com · revision
a1b694b· Accessed Oct 18, 2026
Used for: Model card summary, Is this for you?, Compute & fine-tuning, Setup & usage and 1 more - S003 Pose estimator implementation
Revision, date & where cited
NVlabs/FoundationPose contributors · github.com · revision
a1b694b· Accessed Oct 18, 2026
Used for: Model card summary - S004 Conda build route
Revision, date & where cited
NVlabs/FoundationPose contributors · github.com · revision
a1b694b· Accessed Oct 18, 2026
Used for: Compute & fine-tuning, Inputs, checkpoints, and setup boundaries - S007 NVIDIA source license
Revision, date & where cited
NVlabs/FoundationPose contributors · github.com · revision
a1b694b· Accessed Oct 18, 2026
Used for: Is this for you?, Separate license matrix - S006 Reviewed release and revision
Revision, date & where cited
NVlabs/FoundationPose / GitHub · github.com · revision
a1b694b· commit snapshot; no stable GitHub release / · Accessed Oct 18, 2026
Used for: Overview, What we checked - S008 External weights folder listing
Folder listing reachable; payload content and execution not checked.Revision, date & where cited
FoundationPose authors / Google Drive · drive.google.com · Public folder listing, 2026-10-18 · Accessed Oct 18, 2026
Used for: Checkpoints, Is this for you?, Inputs, checkpoints, and setup boundaries, Separate license matrix - S009 External demo folder listing
Folder listing reachable; payload content and execution not checked.Revision, date & where cited
FoundationPose authors / Google Drive · drive.google.com · Public folder listing, 2026-10-18 · Accessed Oct 18, 2026
Used for: Inputs, checkpoints, and setup boundaries, Separate license matrix - S010 FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects (arXiv:2312.08344)
Unified foundation model for model-based (CAD) and model-free (reference views via neural implicit representation) 6D pose estimation and tracking of novel objects without test-time fine-tuning, trained on large-scale synthetic data.Revision, date & where cited
NVIDIA Research (CVPR 2024 Highlight / arXiv) · arxiv.org · arXiv:2312.08344v2 abstract · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Compute & fine-tuning and 2 more - S011 RefineNet and ScoreNetMultiPair network definitions in NVlabs/FoundationPose
RefineNet combines ResnetBasicBlock feature stages (128/256/512) with TransformerEncoderLayer blocks; ScoreNetMultiPair uses ResnetBasicBlock stages for multi-pair pose hypothesis scoring.Revision, date & where cited
NVlabs/FoundationPose contributors · github.com · revision
a1b694b· learning/models/refine_network.py and score_network.py at · Accessed Nov 21, 2026
Used for: Overview, Model card summary
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Migrated from 0.7.0-draft.5 project to 0.8.0-draft.1 model; added RefineNet/ScoreNetMultiPair architecture, two Drive checkpoint entries, and upstream-reported BOP23 benchmark.