Cosmos-Transfer1
Robotics modelNVIDIA 7B MultiControlNet world-to-world transfer model for robotics Sim2Real and autonomous-vehicle video generation across depth, segmentation, edge, vis/blur, keypoint, LiDAR, and HDMap controls; Apache-2.0 code and gated NVIDIA Open Model License weights, now in limited maintenance as NVIDIA directs users to Cosmos 3. [1] [2] [4]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Robotics Sim2Real and synthetic-data engineers; Autonomous vehicle simulation teams; World-model researchers with multi-GPU datacenter hardware [2] [3] [5]
- Parameters
- 7B base diffusion model + modality ControlNet branches (base_model.pt 14.47 GB [2] [3] [4]
- Embodiments
- Embodiment-agnostic Sim2Real world-to-world transfer (robotic manipulation scene augmentation workflow), Autonomous… [2] [5]
- Source rights
- Named licenses are documented by part. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
Overview
Cosmos-Transfer1 is NVIDIA's 7B conditional video diffusion world-to-world transfer model designed to bridge synthetic-to-real (Sim2Real) visual gaps across robotics and autonomous-vehicle pipelines. Using single-modality ControlNet branches or a MultiControlNet with adaptive spatiotemporal control weights, it conditions generated 720p video rollouts on segmentation, depth, edge, blur/vis, keypoint, or (in the AV sample) LiDAR and HDMap videos alongside text prompts and optional RGB frame prompts. Although the GitHub repository (nvidia-cosmos/cosmos-transfer1) was not archived when checked on 2026-11-21 (reviewed HEAD commit 5005e823dbd4, dated 2026-06-07), its README banner states that it is no longer under active development and will receive only limited maintenance updates as NVIDIA shifts to Cosmos 3. [1] [2] [5]
Model card summary
- Architecture
7B conditional video diffusion world-to-world transfer model using single-modality ControlNet or MultiControlNet with an adaptive spatiotemporal control weight map across segmentation, depth, edge, blur/vis, keypoint, and AV LiDAR/HDMap branches, paired with Cosmos-Tokenize1-CV8x8x8-720p, optional 1-step distilled edge ControlNet, and a 4K upscaler.
- Parameter count
7B base diffusion model + modality ControlNet branches (base_model.pt 14.47 GB; each ControlNet branch ~3.57 GB; edge_control_distilled.pt 16.26 GB; paired with optional 12B prompt upsampler)
- Input modalities
- rgbdepthpoint-cloudlanguage
Accepts text prompts, optional RGB video frame prompts, depth video, segmentation/edge/blur/keypoint control videos, and LiDAR/HDMap control videos in the AV sample.
- Output modalities
- video-frames
Outputs 720p world-transfer RGB video frames (or 4K via Cosmos-Transfer1-7B-4KUpscaler).
- Action space
Spatiotemporal visual control videos (depth, segmentation, edge, blur/vis, keypoint, LiDAR, HDMap) weighted by adaptive spatiotemporal control maps plus text prompts; does not take low-level robot action vectors or output motor commands.
- Control frequency
Not specified as a closed-loop control rate (offline video-to-video transfer; paper notes real-time scaling demonstrated on an NVIDIA GB200 NVL72 rack)
- Trained / supported embodiments
- Embodiment-agnostic Sim2Real world-to-world transfer (robotic manipulation scene augmentation workflow)
- Autonomous vehicles (LiDAR + HDMap single-view and multi-view sample checkpoints)
- Training datasets
- Pretraining video and control datasets are not publicly released
Repository provides pre-training and post-training scripts for user-supplied data, but not the upstream pretraining corpus.
- Interfaces
- PyTorch torchrun CLI (cosmos_transfer1/diffusion/inference/transfer.py)
- JSON ControlNet specification files (--controlnet_specs)
- Docker container workflow
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| nvidia/Cosmos-Transfer1-7B | pytorch | 86.79 GB | gated | ef4dc8e3437a |
| nvidia/Cosmos-Transfer1-7B-Sample-AV | pytorch | 70.7 GB | gated | 9ec22b4a5c46 |
| nvidia/Cosmos-Transfer1-7B-Sample-AV-Single2MultiView | pytorch | 72.12 GB | gated | 85f50fc5ed4d |
| nvidia/Cosmos-Transfer1-7B-4KUpscaler | pytorch | 38.19 GB | gated | 91029686c776 |
Sizes are decimal GB sums of sibling files at the pinned HF revision (Cosmos-Transfer1-7B includes base_model.pt, depth/edge/keypoint/seg/vis/4kupscaler controls, edge_control_distilled.pt, and bundled guardrail weights). All 4 repos are gated='auto' under NVIDIA Open Model License.
Is this for you?
- Use cases
- Robotics Sim2Real scene augmentation (mapping one synthetic manipulation rollout to diverse realistic visual domains)
- Multimodal adaptive spatiotemporal world-to-world transfer (depth, segmentation, edge, blur/vis, keypoint)
- Autonomous vehicle LiDAR and HDMap conditional single-view and multi-view video generation
- 720p-to-4K video super-resolution via Cosmos-Transfer1-7B-4KUpscaler
- Known limitations
- No longer under active development; README states only limited maintenance updates will occur as NVIDIA focuses on Cosmos 3
- Model weights are gated on Hugging Face under the NVIDIA Open Model License, and full checkpoint download requires accepting Meta Llama-Guard-3-8B terms and ~300 GB free storage
- Training and post-training require 8x 80 GB NVIDIA GPUs (H100-80GB or A100-80GB recommended) with Tensor Parallel size 8
- Pretraining datasets are not released; Linux-only installation requires Python 3.12.x and CUDA 12.8 wheel stack
- Safety and limitations
- Generates visual world-transfer videos for Sim2Real augmentation rather than closed-loop robot control commands; supplied documentation does not establish physical-contact accuracy.
- Active development has ended in favor of Cosmos 3 (limited maintenance updates only).
- Model weights are gated under the NVIDIA Open Model License (not OSI-approved), and default guardrails require accepting Meta Llama-Guard-3-8B terms.
- Heavy storage and training compute footprint (~300 GB for full checkpoint download; 8x 80 GB GPUs required for training/post-training).
Compute & fine-tuning
- Inference compute
- Documented hardware: 1+ NVIDIA GPUs via torchrun (with offload flags); real-time scaling reported on NVIDIA GB200 NVL72 rack
- Precision:
bf16 - Edge / consumer capable: No
Exact minimum single-GPU inference VRAM is not specified in the inference guide; training guide specifies 8x 80 GB GPUs (H100-80GB or A100-80GB).
- Fine-tuning
- Supported: Yes
- Methods: Post-training per control modality (Tensor Parallel 8); Pre-training from scratch; Single-step distillation
- Minimum VRAM: 80 GB
- Recipe: Upstream fine-tuning documentation
Training matrix requires 8 NVIDIA GPUs with 80 GB memory each (H100-80GB or A100-80GB recommended) using TP=8.
- Host computer
Linux-only NVIDIA GPU host; inference supports 1 or multiple GPUs via torchrun with --offload_text_encoder_model and --offload_guardrail_models (~300 GB disk for full checkpoint set); pre-training and post-training require 8x 80 GB GPUs (H100-80GB or A100-80GB, TP=8).
- Operating system
- Linux (tested on Ubuntu 20.04, 22.04, and 24.04)
- Dependencies
- Python 3.12.x
- PyTorch 2.7.1 (cu128)
- Transformer Engine 1.13.0
- FlashAttention 2.6.3 and NATTEN 0.21.0
- vLLM 0.9.2 and FlashInfer
- Cosmos-Tokenize1-CV8x8x8-720p, Cosmos-Guardrail1, and Llama-Guard-3-8B
- Skills
- Linux and Conda or Docker (NVIDIA Container Toolkit)
- PyTorch distributed (torchrun) and CUDA 12.8 tooling
- ControlNet / diffusion inference and JSON control-spec configuration
Inferred from INSTALL.md and inference_cosmos_transfer1_7b.md.
[3] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [3] · Model card / documentation [1] [2]
- Clone nvidia-cosmos/cosmos-transfer1 on Linux (Ubuntu 20.04/22.04/24.04) with Python 3.12.x and CUDA 12.8 libraries (libnvrtc)
- Create the Conda environment from cosmos-transfer1.yaml (or build the Dockerfile) and install PyTorch 2.7.1+cu128, Transformer Engine, FlashAttention, NATTEN, and vLLM 0.9.2
- Log in to Hugging Face, accept NVIDIA Open Model License and Llama-Guard-3-8B terms, and run scripts/download_checkpoints.py (~300 GB disk)
- Run torchrun cosmos_transfer1/diffusion/inference/transfer.py with --controlnet_specs and optional offload flags
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| Cosmos-Transfer1 technical report evaluations (arXiv:2503.14492) and distilled edge ControlNet | Upstream reports single-step generation (1 diffusion step vs. 36 steps) for edge_control_distilled.pt, and demonstrates real-time inference scaling on an NVIDIA GB200 NVL72 rack alongside robotics Sim2Real and AV data enrichment evaluations. | Upstream-reported paper and README claims; not independently verified by OSR. |
Project notes
Robotics Sim2Real workflows and upstream-reported capabilities
For robotics, the repository includes a dedicated augmentation workflow mapping a single synthetic robotic manipulation rollout into multiple photorealistic visual variations, alongside a 4K upscaler (Cosmos-Transfer1-7B-4KUpscaler) and a 1-step distilled edge ControlNet (edge_control_distilled.pt, versus 36 diffusion steps in the base model). Unlike action-conditioned controllers, Cosmos-Transfer1 maps visual control streams to RGB video frames rather than outputting low-level robot motor commands, and the reviewed documentation does not establish physical-contact fidelity. Upstream's technical report (arXiv:2503.14492v2) also describes real-time inference scaling demonstrated on an NVIDIA GB200 NVL72 rack. All reported distillation speedups, Sim2Real evaluations, and scaling results are upstream-reported; Open Source Robotics has not run training, fine-tuning, or inference on Cosmos-Transfer1. [2] [4] [5]
Compute, storage, and environment constraints
Installation is Linux-only (tested on Ubuntu 20.04, 22.04, and 24.04) and requires Python 3.12.x with a pinned CUDA 12.8 stack (PyTorch 2.7.1, Transformer Engine, FlashAttention, NATTEN, and vLLM 0.9.2). Downloading the full checkpoint suite requires roughly 300 GB of free disk space (nvidia/Cosmos-Transfer1-7B alone totals 86.79 GB at HEAD across base weights, modality ControlNets, distilled edge weights, and bundled guardrails). Inference supports single- or multi-GPU torchrun execution with --offload_text_encoder_model and --offload_guardrail_models (minimum single-GPU inference VRAM is not quantified upstream), whereas pre-training and post-training explicitly require 8x 80 GB GPUs (H100-80GB or A100-80GB with Tensor Parallel size 8). [3] [4]
What we checked
None. Prepared from public upstream sources without hands-on training, fine-tuning, or inference.
Updates
- Upstream last updated
- Jun 7, 2026 [1]
- Latest release
- Nov 19, 2025 [4]
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- nvidia-cosmos/cosmos-transfer1@5005e823dbd478ad8e51f6bc28a913a13a994b5f; HF Cosmos-Transfer1-7B@ef4dc8e3437a8518dbb92b23aa3cfdaee7d6d54f; HF Cosmos-Transfer1-7B-Sample-AV@9ec22b4a5c46c674ea5df38e909ee63493519d38 · software repository
Separate license matrix
Prominent multimodal Sim2Real world-transfer model with Apache-2.0 inference/training/distillation code and 4 public checkpoints, marked include_partial because weights are gated under the NVIDIA Open Model License, pretraining datasets are unreleased, and active development has shifted to Cosmos 3.
| Layer | Availability / license |
|---|---|
| Code | available · Apache-2.0 Verified Apache-2.0 in root LICENSE file and README. [1] [2] |
| Model weights | available · NVIDIA Open Model License (gated) All 4 Hugging Face repositories are gated='auto' under the NVIDIA Open Model License; Llama Guard 3 filter carries its own license. [2] [4] |
| Training datasets | unavailable · License not fully assessed Upstream pretraining datasets are not publicly released. [2] [3] |
| Documentation | available · Apache-2.0 [1] [2] [3] |
Licensing boundaries and evaluation next steps
Source code for inference, pre-training, post-training, and distillation is licensed under Apache-2.0 (LICENSE). All four Hugging Face model repositories are gated (gated: auto) under the NVIDIA Open Model License, and downloading the full inference stack also requires accepting Meta's Llama-Guard-3-8B terms for input safety filtering. Pretraining datasets are not publicly released. As an editorial evaluation sequence, teams considering Cosmos-Transfer1 should first decide whether limited maintenance (versus migrating to Cosmos 3) fits their support requirements, review the NVIDIA Open Model License and Llama-Guard-3-8B terms, and then consult the pinned installation and inference guides—keeping the documented 8x 80 GB training requirement distinct from the unresolved single-GPU inference memory requirement. [1] [2] [3] [4]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 nvidia-cosmos/cosmos-transfer1 GitHub metadata, HEAD commit, and LICENSE
Default branch main HEAD is 5005e823dbd478ad8e51f6bc28a913a13a994b5f (2026-06-07T17:21:25Z); archived=false; root LICENSE is Apache-2.0.Revision, date & where cited
nvidia-cosmos (GitHub) · github.com · revision
5005e82· GitHub API repos/nvidia-cosmos/cosmos-transfer1 & root LICENSE at 5005e823dbd4 · Accessed Nov 21, 2026
Used for: Overview, Setup & usage, What we checked, Separate license matrix - S002 nvidia-cosmos/cosmos-transfer1 README.md at pinned commit
Documents ControlNet/MultiControlNet modalities (segmentation, depth, edge, blur/vis, keypoint, LiDAR, HDMap), 1-step distilled edge ControlNet (vs 36 steps), 4KUpscaler, robotics augmentation workflow, Apache-2.0 code vs NVIDIA Open Model License weights, and notice that active development moved to Cosmos 3.Revision, date & where cited
nvidia-cosmos (GitHub) · github.com · revision
5005e82· README.md at commit 5005e823dbd4 · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 5 more - S003 Cosmos-Transfer1 INSTALL.md, inference_cosmos_transfer1_7b.md, and training_cosmos_transfer_7b.md
Linux-only (Ubuntu 20.04/22.04/24.04), Python 3.12.x, PyTorch 2.7.1+cu128, ~300 GB disk for full checkpoint download, Llama-Guard-3-8B acceptance required, and 8x 80 GB GPUs (H100-80GB/A100-80GB, TP=8) for pre-training/post-training.Revision, date & where cited
nvidia-cosmos (GitHub) · github.com · revision
5005e82· INSTALL.md, examples/inference_cosmos_transfer1_7b.md, examples/training_cosmos_transfer_7b.md · Accessed Nov 21, 2026
Used for: Model card summary, Is this for you?, Compute & fine-tuning, Setup & usage and 2 more - S004 Hugging Face API metadata for 4 nvidia/Cosmos-Transfer1-7B* checkpoints
Verified pinned SHAs, sibling file sizes (38.19 GB to 86.79 GB at HEAD), gated='auto', and license_name='nvidia-open-model-license' across all 4 checkpoints.Revision, date & where cited
nvidia (Hugging Face) · huggingface.co · HF API /api/models/nvidia/Cosmos-Transfer1-7B* (4 repos) · Accessed Nov 21, 2026
Used for: Model card summary, Checkpoints, Is this for you?, Robotics Sim2Real workflows and upstream-reported capabilities and 3 more - S005 Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control (arXiv:2503.14492v2)
Describes adaptive spatiotemporal multimodal control for Sim2Real robotics and autonomous-vehicle data enrichment, plus real-time inference scaling demonstrated on an NVIDIA GB200 NVL72 rack.Revision, date & where cited
arXiv (NVIDIA) · arxiv.org · arXiv abstract 2503.14492v2 (2025-04-01) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Compute & fine-tuning and 2 more
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from GitHub commit 5005e823dbd4, arXiv:2503.14492v2, and 4 verified Hugging Face checkpoints.