Open Source RoboticsRobots, software & learning

FoundationPose

Robotics model

Research implementation for 6D object pose estimation and tracking from RGB-D inputs, using a CAD model or reference views. [1]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Researchers evaluating object-pose perception [1]
Parameters
Not stated in upstream README or arXiv:2312.08344 abstract (two separate PyTorch networks: RefineNet refiner and… [1] [10] [11]
Embodiments
Embodiment-agnostic perception module (RGB-D camera input [1]
Source rights
Some licenses are unresolved. Includes noncommercial terms. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware
License
NVIDIA Source Code License (noncommercial research/evaluation) (noncommercial) · license not fully assessed for model weights, datasets · Details by part

Requirements · Research details and dates

On this page

Overview

FoundationPose (NVlabs/FoundationPose, CVPR 2024 Highlight, arXiv:2312.08344) is a unified 6D object pose estimation and tracking model supporting both model-based (CAD mesh) and model-free (few-shot reference views via a neural implicit representation) setups without test-time fine-tuning. At pinned commit a1b694b83e633c2cb6115b9063d940a687759392, its core networks combine ResNet feature stages with Transformer encoder layers in a pose refiner (RefineNet) and multi-pair pose scorer (ScoreNetMultiPair). [1] [6] [10] [11]

Model card summary

Architecture

Unified 6D pose estimation and tracking architecture pairing a ResNet + Transformer pose refiner (RefineNet) and multi-pair pose scorer (ScoreNetMultiPair), bridged to model-free few-shot operation via a neural implicit representation for novel view synthesis.

[1] [10] [11]

Parameter count

Not stated in upstream README or arXiv:2312.08344 abstract (two separate PyTorch networks: RefineNet refiner and ScoreNetMultiPair scorer)

Weight binaries on Google Drive were not downloaded to count parameters.

[1] [10] [11] · Editorial assessment

Input modalities
rgbd

RGB-D frames plus camera intrinsics, initial object segmentation mask, and either a 3D CAD mesh or reference views.

[1] [2]

Output modalities
6d-pose

Per-frame 4x4 SE(3) object-in-camera 6D pose matrices.

[1] [2]

Action space

Perception-only model (no robot action output); produces per-frame 4x4 SE(3) object-in-camera 6D pose matrices requiring an external robot controller or grasp planner.

[1] [2]

Trained / supported embodiments
  • Embodiment-agnostic perception module (RGB-D camera input; upstream links separate NVIDIA-ISAAC-ROS/isaac_ros_pose_estimation for ROS 2 / TensorRT C++ deployment)

[1]

Training datasets
  • FoundationPose synthetic dataset rendered from GSO and Objaverse 3D assets with domain randomization (public release excludes Stable-Diffusion texture-augmented data and weights trained on it)

[1]

Interfaces
  • Python API (FoundationPose / run_demo.py)
  • Docker / Conda CLI
  • Isaac ROS Pose Estimation (separate ROS 2 / TensorRT repo linked by upstream)

[1] [2] [3]

Checkpoints

CheckpointFormatSizeAccessPinned revision
FoundationPose Refiner (2023-10-28-18-33-37)pytorchNot statedopen2023-10-28-18-33-37
FoundationPose Scorer (2024-01-11-20-02-45)pytorchNot statedopen2024-01-11-20-02-45

Public Google Drive folder listing reachable; payloads were not downloaded, so exact file sizes are unknown. Public weights omit diffusion-augmented training data.

[1] [8]

Is this for you?

Use cases
  • Zero-shot 6D object pose estimation from RGB-D and a CAD mesh (model-based)
  • Few-shot 6D pose estimation from reference views via neural implicit rendering (model-free)
  • Frame-to-frame 6D object pose tracking for robotics manipulation and AR

[1] [2] [10]

Known limitations
  • Repository license limits use to noncommercial research or evaluation.
  • External weights and data were not downloaded or validated in this review.
  • run_demo.py removes existing debug-directory contents at startup.
  • The released model excludes diffusion-augmented training data and weights.

[1] [2] [7]

Safety and limitations
  • Repository NVIDIA Source Code License restricts use to noncommercial research or evaluation.
  • Released weights exclude Stable-Diffusion texture-augmented training data due to LAION/Stable-Diffusion legal restrictions; upstream notes slight performance degradation is expected.
  • run_demo.py removes existing contents of --debug_dir at startup via a shell command; always pass a disposable path without shell metacharacters.
  • External Google Drive weights and demo payloads were not downloaded, executed, or benchmarked by Open Source Robotics.

[1] [2] [7] [8]

Compute & fine-tuning

Inference compute
  • Documented hardware: NVIDIA CUDA GPU (PyTorch, PyTorch3D, NVDiffRast; separate Isaac ROS repo offers TensorRT C++ inference)
  • Edge / consumer capable: No

Requires CUDA GPU for PyTorch3D/NVDiffRast rendering; exact minimum VRAM and latency_ms not stated in README.

[1] [4] · Editorial assessment

Fine-tuning
  • Supported: No

Designed for zero-shot test-time application to novel objects without fine-tuning; training scripts are not part of the documented README workflow.

[1] [10]

Host computer

CUDA-capable NVIDIA GPU required for PyTorch, PyTorch3D, NVDiffRast, and compiled C++/CUDA extensions; exact minimum VRAM not stated in README.

[1] [4]

Operating system
  • Linux

Docker and Conda Linux build routes documented; Windows community issue linked.

[1] [4] · Editorial assessment

Dependencies
  • CUDA-capable NVIDIA environment
  • PyTorch, PyTorch3D and NVDiffRast
  • Compiled extension and external scorer/refiner weights
  • RGB-D demo inputs and a mesh

[1] [2]

Setup & usage

Open the upstream getting-started guide [1]

  • Prepare the documented CUDA environment and build extensions.
  • Obtain the named scorer/refiner weights and demo data.
  • Provide a mesh, RGB-D sequence, camera intrinsics and initial object mask.
  • Run run_demo.py with a disposable debug directory and inspect saved pose matrices.

Editorial assessment from cited documentation; no runtime testing.

[1] [2] · Editorial assessment

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
BOP23 Core Datasets Leaderboard (Model-based Novel Object Pose Estimation)Ranked #1 as of March 2024 per upstream README (CVPR 2024 Highlight, arXiv:2312.08344)Upstream-reported leaderboard standing; public weights exclude diffusion-augmented training data so upstream notes slight performance degradation is expected. OSR has not reproduced BOP evaluations.

[1] [10]

Project notes

Inputs, checkpoints, and setup boundaries

FoundationPose is a perception-only model: the mesh-based demo run_demo.py takes RGB-D frames, camera intrinsics, an initial object segmentation mask, and a 3D mesh, and outputs per-frame 4×4 SE(3) object-in-camera pose matrices without generating robot control commands. Model-free operation is handled separately from run_demo.py: the README documents first training a per-object Neural Object Field from reference views (object-representation preparation distinct from fine-tuning the pose networks) and then invoking the dataset-runner workflow with the reconstructed mesh and reference-view arguments, noting that build_all_conda.sh omits the BundleSDF CUDA operations required for that model-free path. The README links two external PyTorch weight directories on Google Drive—2023-10-28-18-33-37 (refiner) and 2024-01-11-20-02-45 (scorer)—plus demo data and synthetic GSO/Objaverse training data. The Google Drive folder listings were verified reachable, but binary payloads were not downloaded, so exact parameter counts and checkpoint sizes are not established here. [1] [2] [4] [8] [9]

Installation requires a CUDA-capable NVIDIA GPU with PyTorch, PyTorch3D, NVDiffRast, and compiled C++/CUDA extensions via either the documented Docker or Conda route (upstream also links a separate NVIDIA-ISAAC-ROS/isaac_ros_pose_estimation repo for ROS 2 / TensorRT C++ deployment). Note that run_demo.py deletes existing --debug_dir contents at startup via a shell command, so users should pass a disposable directory path without spaces or shell metacharacters. [1] [2] [4]

Upstream-reported benchmarks and release omissions

The README reports a #1 standing on the BOP23 core datasets leaderboard for model-based novel object pose estimation (as of March 2024). However, upstream explicitly notes that Stable-Diffusion texture-augmented training data and weights trained on it could not be released due to LAION/Stable-Diffusion legal restrictions, and that slight performance degradation is expected for the public checkpoints. All benchmark results are upstream-reported; Open Source Robotics has not installed dependencies, downloaded weights, run inference, or reproduced BOP evaluations. [1] [10]

What we checked

Source researched only; no installation, training, benchmark or physical hardware test performed. No affiliate links or reused media.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Apr 29, 2026 [6]
Latest release
Not established
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
commit snapshot; no stable GitHub release / a1b694b83e633c2cb6115b9063d940a687759392 · software repository

Separate license matrix

Include the inspectable robotics pose-estimation implementation under a restricted research scope; distinguish source availability from unverified external weights and production rights.

LayerAvailability / license
Codeavailable · NVIDIA Source Code License (noncommercial research/evaluation)
Repository materials only; external components retain their own terms. [7]
Model weightsavailable · License not fully assessed
Public Drive folder listing retrieved successfully; payloads not downloaded or validated and artifact-specific terms remain unverified. [1] [8]
Training datasetsavailable · License not fully assessed
Public Drive folder listing retrieved successfully; payloads not downloaded or validated and artifact-specific terms remain unverified. [1] [9]
Documentationavailable · NVIDIA Source Code License (noncommercial research/evaluation)
Repository materials only; external components retain their own terms. [7]

Noncommercial research license

The repository LICENSE (NVIDIA Source Code License) restricts use to noncommercial research or evaluation—not permissive commercial production deployment—and the README states that code and data are released under that license. However, this review has not verified artifact-specific license terms accompanying the external Google Drive weights or dataset payloads, and third-party dependencies retain their own separate licensing terms. [1] [7] [8] [9]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Migrated from 0.7.0-draft.5 project to 0.8.0-draft.1 model; added RefineNet/ScoreNetMultiPair architecture, two Drive checkpoint entries, and upstream-reported BOP23 benchmark.