Open Source RoboticsRobots, software & learning

Cosmos-Transfer1

Robotics model

NVIDIA 7B MultiControlNet world-to-world transfer model for robotics Sim2Real and autonomous-vehicle video generation across depth, segmentation, edge, vis/blur, keypoint, LiDAR, and HDMap controls; Apache-2.0 code and gated NVIDIA Open Model License weights, now in limited maintenance as NVIDIA directs users to Cosmos 3. [1] [2] [4]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Robotics Sim2Real and synthetic-data engineers; Autonomous vehicle simulation teams; World-model researchers with multi-GPU datacenter hardware [2] [3] [5]
Parameters
7B base diffusion model + modality ControlNet branches (base_model.pt 14.47 GB [2] [3] [4]
Embodiments
Embodiment-agnostic Sim2Real world-to-world transfer (robotic manipulation scene augmentation workflow), Autonomous… [2] [5]
Source rights
Named licenses are documented by part. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware

Requirements · Research details and dates

On this page

Overview

Cosmos-Transfer1 is NVIDIA's 7B conditional video diffusion world-to-world transfer model designed to bridge synthetic-to-real (Sim2Real) visual gaps across robotics and autonomous-vehicle pipelines. Using single-modality ControlNet branches or a MultiControlNet with adaptive spatiotemporal control weights, it conditions generated 720p video rollouts on segmentation, depth, edge, blur/vis, keypoint, or (in the AV sample) LiDAR and HDMap videos alongside text prompts and optional RGB frame prompts. Although the GitHub repository (nvidia-cosmos/cosmos-transfer1) was not archived when checked on 2026-11-21 (reviewed HEAD commit 5005e823dbd4, dated 2026-06-07), its README banner states that it is no longer under active development and will receive only limited maintenance updates as NVIDIA shifts to Cosmos 3. [1] [2] [5]

Model card summary

Architecture

7B conditional video diffusion world-to-world transfer model using single-modality ControlNet or MultiControlNet with an adaptive spatiotemporal control weight map across segmentation, depth, edge, blur/vis, keypoint, and AV LiDAR/HDMap branches, paired with Cosmos-Tokenize1-CV8x8x8-720p, optional 1-step distilled edge ControlNet, and a 4K upscaler.

[2] [3] [5]

Base model

Cosmos-Transfer1-7B base diffusion model + modality ControlNet branches

[2] [3]

Parameter count

7B base diffusion model + modality ControlNet branches (base_model.pt 14.47 GB; each ControlNet branch ~3.57 GB; edge_control_distilled.pt 16.26 GB; paired with optional 12B prompt upsampler)

[2] [3] [4]

Input modalities
rgbdepthpoint-cloudlanguage

Accepts text prompts, optional RGB video frame prompts, depth video, segmentation/edge/blur/keypoint control videos, and LiDAR/HDMap control videos in the AV sample.

[2] [5]

Output modalities
video-frames

Outputs 720p world-transfer RGB video frames (or 4K via Cosmos-Transfer1-7B-4KUpscaler).

[2] [5]

Action space

Spatiotemporal visual control videos (depth, segmentation, edge, blur/vis, keypoint, LiDAR, HDMap) weighted by adaptive spatiotemporal control maps plus text prompts; does not take low-level robot action vectors or output motor commands.

[2] [5]

Control frequency

Not specified as a closed-loop control rate (offline video-to-video transfer; paper notes real-time scaling demonstrated on an NVIDIA GB200 NVL72 rack)

[5]

Trained / supported embodiments
  • Embodiment-agnostic Sim2Real world-to-world transfer (robotic manipulation scene augmentation workflow)
  • Autonomous vehicles (LiDAR + HDMap single-view and multi-view sample checkpoints)

[2] [5]

Training datasets
  • Pretraining video and control datasets are not publicly released

Repository provides pre-training and post-training scripts for user-supplied data, but not the upstream pretraining corpus.

[2] [3]

Interfaces
  • PyTorch torchrun CLI (cosmos_transfer1/diffusion/inference/transfer.py)
  • JSON ControlNet specification files (--controlnet_specs)
  • Docker container workflow

[3]

Checkpoints

CheckpointFormatSizeAccessPinned revision
nvidia/Cosmos-Transfer1-7Bpytorch86.79 GBgatedef4dc8e3437a
nvidia/Cosmos-Transfer1-7B-Sample-AVpytorch70.7 GBgated9ec22b4a5c46
nvidia/Cosmos-Transfer1-7B-Sample-AV-Single2MultiViewpytorch72.12 GBgated85f50fc5ed4d
nvidia/Cosmos-Transfer1-7B-4KUpscalerpytorch38.19 GBgated91029686c776

Sizes are decimal GB sums of sibling files at the pinned HF revision (Cosmos-Transfer1-7B includes base_model.pt, depth/edge/keypoint/seg/vis/4kupscaler controls, edge_control_distilled.pt, and bundled guardrail weights). All 4 repos are gated='auto' under NVIDIA Open Model License.

[2] [4]

Is this for you?

Use cases
  • Robotics Sim2Real scene augmentation (mapping one synthetic manipulation rollout to diverse realistic visual domains)
  • Multimodal adaptive spatiotemporal world-to-world transfer (depth, segmentation, edge, blur/vis, keypoint)
  • Autonomous vehicle LiDAR and HDMap conditional single-view and multi-view video generation
  • 720p-to-4K video super-resolution via Cosmos-Transfer1-7B-4KUpscaler

[2] [5]

Known limitations
  • No longer under active development; README states only limited maintenance updates will occur as NVIDIA focuses on Cosmos 3
  • Model weights are gated on Hugging Face under the NVIDIA Open Model License, and full checkpoint download requires accepting Meta Llama-Guard-3-8B terms and ~300 GB free storage
  • Training and post-training require 8x 80 GB NVIDIA GPUs (H100-80GB or A100-80GB recommended) with Tensor Parallel size 8
  • Pretraining datasets are not released; Linux-only installation requires Python 3.12.x and CUDA 12.8 wheel stack

[2] [3] [4]

Safety and limitations
  • Generates visual world-transfer videos for Sim2Real augmentation rather than closed-loop robot control commands; supplied documentation does not establish physical-contact accuracy.
  • Active development has ended in favor of Cosmos 3 (limited maintenance updates only).
  • Model weights are gated under the NVIDIA Open Model License (not OSI-approved), and default guardrails require accepting Meta Llama-Guard-3-8B terms.
  • Heavy storage and training compute footprint (~300 GB for full checkpoint download; 8x 80 GB GPUs required for training/post-training).

[2] [3] [4]

Compute & fine-tuning

Inference compute
  • Documented hardware: 1+ NVIDIA GPUs via torchrun (with offload flags); real-time scaling reported on NVIDIA GB200 NVL72 rack
  • Precision: bf16
  • Edge / consumer capable: No

Exact minimum single-GPU inference VRAM is not specified in the inference guide; training guide specifies 8x 80 GB GPUs (H100-80GB or A100-80GB).

[3] [5]

Fine-tuning
  • Supported: Yes
  • Methods: Post-training per control modality (Tensor Parallel 8); Pre-training from scratch; Single-step distillation
  • Minimum VRAM: 80 GB
  • Recipe: Upstream fine-tuning documentation

Training matrix requires 8 NVIDIA GPUs with 80 GB memory each (H100-80GB or A100-80GB recommended) using TP=8.

[2] [3]

Host computer

Linux-only NVIDIA GPU host; inference supports 1 or multiple GPUs via torchrun with --offload_text_encoder_model and --offload_guardrail_models (~300 GB disk for full checkpoint set); pre-training and post-training require 8x 80 GB GPUs (H100-80GB or A100-80GB, TP=8).

[3]

Operating system
  • Linux (tested on Ubuntu 20.04, 22.04, and 24.04)

[3]

Dependencies
  • Python 3.12.x
  • PyTorch 2.7.1 (cu128)
  • Transformer Engine 1.13.0
  • FlashAttention 2.6.3 and NATTEN 0.21.0
  • vLLM 0.9.2 and FlashInfer
  • Cosmos-Tokenize1-CV8x8x8-720p, Cosmos-Guardrail1, and Llama-Guard-3-8B

[3]

Skills
  • Linux and Conda or Docker (NVIDIA Container Toolkit)
  • PyTorch distributed (torchrun) and CUDA 12.8 tooling
  • ControlNet / diffusion inference and JSON control-spec configuration

Inferred from INSTALL.md and inference_cosmos_transfer1_7b.md.

[3] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [3] · Model card / documentation [1] [2]

  • Clone nvidia-cosmos/cosmos-transfer1 on Linux (Ubuntu 20.04/22.04/24.04) with Python 3.12.x and CUDA 12.8 libraries (libnvrtc)
  • Create the Conda environment from cosmos-transfer1.yaml (or build the Dockerfile) and install PyTorch 2.7.1+cu128, Transformer Engine, FlashAttention, NATTEN, and vLLM 0.9.2
  • Log in to Hugging Face, accept NVIDIA Open Model License and Llama-Guard-3-8B terms, and run scripts/download_checkpoints.py (~300 GB disk)
  • Run torchrun cosmos_transfer1/diffusion/inference/transfer.py with --controlnet_specs and optional offload flags

[3]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
Cosmos-Transfer1 technical report evaluations (arXiv:2503.14492) and distilled edge ControlNetUpstream reports single-step generation (1 diffusion step vs. 36 steps) for edge_control_distilled.pt, and demonstrates real-time inference scaling on an NVIDIA GB200 NVL72 rack alongside robotics Sim2Real and AV data enrichment evaluations.Upstream-reported paper and README claims; not independently verified by OSR.

[2] [5]

Project notes

Robotics Sim2Real workflows and upstream-reported capabilities

For robotics, the repository includes a dedicated augmentation workflow mapping a single synthetic robotic manipulation rollout into multiple photorealistic visual variations, alongside a 4K upscaler (Cosmos-Transfer1-7B-4KUpscaler) and a 1-step distilled edge ControlNet (edge_control_distilled.pt, versus 36 diffusion steps in the base model). Unlike action-conditioned controllers, Cosmos-Transfer1 maps visual control streams to RGB video frames rather than outputting low-level robot motor commands, and the reviewed documentation does not establish physical-contact fidelity. Upstream's technical report (arXiv:2503.14492v2) also describes real-time inference scaling demonstrated on an NVIDIA GB200 NVL72 rack. All reported distillation speedups, Sim2Real evaluations, and scaling results are upstream-reported; Open Source Robotics has not run training, fine-tuning, or inference on Cosmos-Transfer1. [2] [4] [5]

Compute, storage, and environment constraints

Installation is Linux-only (tested on Ubuntu 20.04, 22.04, and 24.04) and requires Python 3.12.x with a pinned CUDA 12.8 stack (PyTorch 2.7.1, Transformer Engine, FlashAttention, NATTEN, and vLLM 0.9.2). Downloading the full checkpoint suite requires roughly 300 GB of free disk space (nvidia/Cosmos-Transfer1-7B alone totals 86.79 GB at HEAD across base weights, modality ControlNets, distilled edge weights, and bundled guardrails). Inference supports single- or multi-GPU torchrun execution with --offload_text_encoder_model and --offload_guardrail_models (minimum single-GPU inference VRAM is not quantified upstream), whereas pre-training and post-training explicitly require 8x 80 GB GPUs (H100-80GB or A100-80GB with Tensor Parallel size 8). [3] [4]

What we checked

None. Prepared from public upstream sources without hands-on training, fine-tuning, or inference.

Updates

Upstream last updated
Jun 7, 2026 [1]
Latest release
Nov 19, 2025 [4]
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
nvidia-cosmos/cosmos-transfer1@5005e823dbd478ad8e51f6bc28a913a13a994b5f; HF Cosmos-Transfer1-7B@ef4dc8e3437a8518dbb92b23aa3cfdaee7d6d54f; HF Cosmos-Transfer1-7B-Sample-AV@9ec22b4a5c46c674ea5df38e909ee63493519d38 · software repository

Separate license matrix

Prominent multimodal Sim2Real world-transfer model with Apache-2.0 inference/training/distillation code and 4 public checkpoints, marked include_partial because weights are gated under the NVIDIA Open Model License, pretraining datasets are unreleased, and active development has shifted to Cosmos 3.

LayerAvailability / license
Codeavailable · Apache-2.0
Verified Apache-2.0 in root LICENSE file and README. [1] [2]
Model weightsavailable · NVIDIA Open Model License (gated)
All 4 Hugging Face repositories are gated='auto' under the NVIDIA Open Model License; Llama Guard 3 filter carries its own license. [2] [4]
Training datasetsunavailable · License not fully assessed
Upstream pretraining datasets are not publicly released. [2] [3]
Documentationavailable · Apache-2.0
[1] [2] [3]

Licensing boundaries and evaluation next steps

Source code for inference, pre-training, post-training, and distillation is licensed under Apache-2.0 (LICENSE). All four Hugging Face model repositories are gated (gated: auto) under the NVIDIA Open Model License, and downloading the full inference stack also requires accepting Meta's Llama-Guard-3-8B terms for input safety filtering. Pretraining datasets are not publicly released. As an editorial evaluation sequence, teams considering Cosmos-Transfer1 should first decide whether limited maintenance (versus migrating to Cosmos 3) fits their support requirements, review the NVIDIA Open Model License and Llama-Guard-3-8B terms, and then consult the pinned installation and inference guides—keeping the documented 8x 80 GB training requirement distinct from the unresolved single-GPU inference memory requirement. [1] [2] [3] [4]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from GitHub commit 5005e823dbd4, arXiv:2503.14492v2, and 4 verified Hugging Face checkpoints.