CogACT
Robotics modelComponentized Vision-Language-Action model family from Microsoft Research Asia pairing a 7B Prismatic VLM (DINOv2 + SigLIP + Llama-2) with specialized Diffusion Transformer action modules (DiT-Small, DiT-Base, DiT-Large) and MIT-licensed code and weights. [1] [2] [3]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Robotics researchers studying diffusion-head VLA architectures; Manipulation practitioners evaluating SimplerEnv or fine-tuning on RLDS datasets [1] [3]
- Embodiments
- Google Robot (SimplerEnv Fractal / fractal20220817_data), WidowX (SimplerEnv Bridge / BridgeData V2), Open… [1] [3]
- Source rights
- Named licenses are documented by part. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
Overview
CogACT (microsoft/CogACT, arXiv:2411.19650) is a Vision-Language-Action model family from Microsoft Research Asia that decouples high-level visual-language cognition from continuous robot action synthesis. Instead of quantizing actions directly into VLM tokens, CogACT extracts a single cognition token from a 7B Prismatic VLM backbone (DINOv2 ViT-L/14 + SigLIP ViT-So400M/14 + Llama-2) and conditions a specialized Diffusion Transformer (DiT-S, DiT-B, or DiT-L) action head trained on an Open X-Embodiment (oxe_magic_soup_plus_minus) RLDS mixture. [1] [3]
Model card summary
- Architecture
Componentized VLA combining a Prismatic VLM backbone (DINOv2 ViT-L/14 + SigLIP ViT-So400M/14 vision encoders and Llama-2 7B) emitting a single cognition token to condition a Diffusion Transformer action module (DiT-S, DiT-B, or DiT-L) sampled via DDIM (10 steps, CFG).
- Base model
openvla/openvla-7b-prismatic (Prismatic VLM: DINOv2 ViT-L/14 + SigLIP ViT-So400M/14 + Llama-2 7B)
README documents initializing from openvla/openvla-7b-prismatic or PrismaticVLM.
- Parameter count
~7.6B–7.9B total (7B Prismatic VLM backbone + DiT-S / DiT-B / DiT-L action head; exact scalar param count not exposed in .pt HF metadata; FP32 checkpoints are 28.14 GB, 28.43 GB, and 29.24 GB)
Checkpoints are raw PyTorch .pt files without HF safetensors parameter-count metadata; size scaling across Small (28.14 GB), Base (28.43 GB), and Large (29.24 GB) reflects the DiT action module.
- Output modalities
- ee-pose-deltagripper-command
7-DoF end-effector deltas (x, y, z, roll, pitch, yaw, gripper open/close).
- Action space
Continuous 16-step action chunks of 7-DoF normalized end-effector deltas (shape [16, 7], future_action_window_size=15), sampled with 10-step DDIM and classifier-free guidance (cfg_scale 1.5–7), optionally combined via Adaptive Action Ensemble.
- Control frequency
~5.5 Hz on a single NVIDIA A6000 in bfloat16 with Adaptive Action Ensemble (181 ms per 16-action inference; up to k times higher with k-step open-loop chunk execution)
- Trained / supported embodiments
- Google Robot (SimplerEnv Fractal / fractal20220817_data)
- WidowX (SimplerEnv Bridge / BridgeData V2)
- Open X-Embodiment 7-DoF EEF-delta manipulators
- Custom 7-DoF manipulators via RLDS fine-tuning
- Interfaces
- Python load_vla / predict_action / predict_action_batch API
- HTTP REST inference server (scripts/deploy.py, /api/inference)
- SimplerEnv CogACTInference policy wrapper
- torchrun FSDP training CLI (scripts/train.py)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| CogACT/CogACT-Small | pytorch | 28.14 GB | open | ca302a300542 |
| CogACT/CogACT-Base | pytorch | 28.43 GB | open | 6550bf0992f1 |
| CogACT/CogACT-Large | pytorch | 29.24 GB | open | 0a7e03ce8a3c |
Three ungated FP32 PyTorch .pt checkpoints verified via Hugging Face API.
Is this for you?
- Use cases
- Zero-shot single-camera RGB manipulation inference on Open X-Embodiment setups (e.g., Google Robot Fractal, WidowX Bridge)
- FSDP full fine-tuning on custom RLDS manipulation datasets
- HTTP server-client real-robot deployment with Adaptive Action Ensemble
- Known limitations
- Published Hugging Face checkpoints are single FP32 PyTorch .pt files (~28.1–29.2 GB each) requiring ~30 GB GPU memory unless VLM weights are cast to bfloat16 after loading
- Single-view RGB input only (no proprioceptive state or multi-camera fusion in the released checkpoints)
- Upstream recommends multi-GPU FSDP full fine-tuning (8x A100 example) rather than LoRA
- Increasing open-loop action chunk execution steps k (up to 16) trades accuracy for higher control frequency
- Safety and limitations
- Checkpoints are distributed as raw PyTorch .pt pickle files (~28–29 GB); users should verify checkpoint integrity and trust before deserializing.
- Single-view RGB policy without proprioception or collision/safety guardrails; real-world deployment requires dataset-specific fine-tuning and robot-side safety limits.
- Built on Prismatic / Llama-2 7B backbone provenance alongside MIT-licensed CogACT code and checkpoints.
Compute & fine-tuning
- Inference compute
- Minimum VRAM: 30 GB
- Documented hardware: NVIDIA RTX A6000 (bfloat16)
- Precision:
fp32 / bf16 - Reported latency: 181 ms
- Edge / consumer capable: No
README notes ~30 GB memory in FP32 with optional bfloat16 VLM casting; 181 ms average latency on A6000 in bfloat16 (vs 307 ms for single-step OpenVLA on the same device).
- Fine-tuning
- Supported: Yes
- Methods: PyTorch FSDP full fine-tuning via scripts/train.py (~30 epochs recommended); Pretraining from openvla/openvla-7b-prismatic or PrismaticVLM
- Recipe: Upstream fine-tuning documentation
Upstream explicitly recommends full FSDP fine-tuning over LoRA and provides an 8x A100 command; exact minimum single-GPU VRAM is not stated.
- Host computer
Inference requires ~30 GB GPU memory in FP32 (optional bfloat16 VLM casting supported; benchmarked on a single NVIDIA A6000); full FSDP fine-tuning example uses 1 node with 8x A100 GPUs.
- Operating system
- Linux
Standard Linux conda, CUDA 12, flash-attn, and FSDP workflow documented.
[1] · Editorial assessment
- Dependencies
- Python >= 3.8 (3.10 recommended)
- PyTorch >= 2.2.0 and CUDA >= 12.0
- flash-attn==2.5.5 (optional, required for training)
- SimplerEnv (for simulation evaluation)
- requests (for standalone HTTP deployment client)
- Skills
- Python 3.10 and PyTorch / CUDA
- PyTorch FSDP multi-GPU training and RLDS dataset conversion
- HTTP requests client integration for robot control
Inferred from documented conda, FSDP, RLDS, and HTTP deployment instructions.
[1] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [1]
- Create a Python 3.10 conda environment with PyTorch >= 2.2.0 and CUDA >= 12.0
- Clone microsoft/CogACT and run pip install -e . (or pip install -e .[train] plus flash-attn==2.5.5 for training)
- Load CogACT/CogACT-Small, CogACT-Base, or CogACT-Large via load_vla with matching action_model_type (DiT-S, DiT-B, or DiT-L)
- Run model.predict_action with DDIM sampling (10 steps, cfg_scale 1.5) or serve via scripts/deploy.py on port 5500
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| CogACT-Base inference latency vs OpenVLA (NVIDIA A6000, bfloat16) | 181 ms for 16 generated actions (~5.5 Hz with Adaptive Action Ensemble) vs 307 ms for 1 action on OpenVLA | Upstream-reported timing averaged over 100 calls on a single A6000; full SIMPLER and real-robot success tables are in arXiv:2411.19650 and not independently verified by OSR. |
Project notes
Checkpoint family and 16-step DDIM action chunks
Microsoft Research Asia publishes three ungated FP32 PyTorch .pt checkpoints on Hugging Face: CogACT/CogACT-Small (28.14 GB, DiT-S), CogACT/CogACT-Base (28.43 GB, DiT-B), and CogACT/CogACT-Large (29.24 GB, DiT-L). Each takes a single-view RGB image and an English instruction (no proprioceptive state input) and generates a 16-step chunk of 7-DoF normalized end-effector deltas (x, y, z, roll, pitch, yaw, gripper) using 10-step DDIM sampling with classifier-free guidance (cfg_scale 1.5–7). Predicted chunks are unnormalized per dataset statistics (unnorm_key, e.g., fractal20220817_data) and optionally blended across steps with the repository's Adaptive Action Ensemble. Because checkpoints are raw .pt files rather than safetensors, exact scalar parameter counts per DiT variant are not exposed in Hugging Face API metadata, and users should verify file integrity before deserializing. [1] [3]
Compute footprint, deployment server, and upstream timing
Loading a CogACT checkpoint in FP32 uses ~30 GB GPU memory, though model.vlm = model.vlm.to(torch.bfloat16) reduces VLM memory footprint. On a single NVIDIA A6000 in bfloat16, upstream reports 181 ms average inference latency for CogACT-Base (~5.5 Hz with Adaptive Action Ensemble, generating 16 actions per call) compared with 307 ms for single-step autoregressive OpenVLA on the same device; executing k steps open-loop scales effective rate up to k× at the cost of accuracy over longer horizons. For real-robot deployment, scripts/deploy.py exposes an HTTP /api/inference endpoint usable from a minimal Python requests client, and sim_cogact/ integrates with SimplerEnv. For adaptation, upstream explicitly recommends PyTorch FSDP full fine-tuning (~30 epochs; 8× A100 command documented) over LoRA, and does not specify a minimum single-GPU fine-tuning VRAM. All latency and benchmark figures are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [3]
What we checked
None. Prepared from public upstream sources and Hugging Face API metadata without hands-on training, fine-tuning, or inference.
Updates
- Upstream last updated
- Oct 30, 2025 [1]
- Latest release
- Dec 4, 2024 [3]
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- microsoft/CogACT@b174a1b86deedfab4d198d935207e7bb0527994e; HF CogACT/CogACT-Base@6550bf0992f162fc5d74f14ffee30771a9433363 · combined hardware and software repository
Separate license matrix
MIT-licensed diffusion-head VLA family from Microsoft Research Asia with three ungated Hugging Face checkpoints (Small, Base, Large), FSDP training code, and public Open X-Embodiment training recipe.
| Layer | Availability / license |
|---|---|
| Code | available · MIT Verified MIT License (Microsoft Corporation) in root LICENSE file. [1] [2] |
| Model weights | available · MIT README and all three HF model cards declare MIT license for code and model weights. [1] [3] |
| Training datasets | available · Mixed (Open X-Embodiment RLDS mixture; per-dataset upstream licenses apply) [1] [3] |
| Documentation | available · MIT [1] [2] |
MIT licensing and upstream dataset provenance
The microsoft/CogACT repository (verified in the root LICENSE file) and all three Hugging Face model cards declare the MIT License for code and model weights. Under the supplied MIT License text, users may use, copy, modify, merge, publish, distribute, sublicense, and sell copies provided the copyright and permission notice are retained in all copies or substantial portions, with the software supplied "as is" and without warranty. Pretraining uses a subset of the public Open X-Embodiment RLDS corpus (oxe_magic_soup_plus_minus), where individual constituent datasets retain their own upstream licenses, and training-from-scratch recipes initialize from openvla/openvla-7b-prismatic (a Llama-2 7B-based backbone). This review verifies CogACT's MIT declarations but does not resolve whether underlying base-model license obligations also apply; obtain and review the applicable base-model license texts before finalizing compliance for redistribution or commercial deployment. [1] [2] [3]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 microsoft/CogACT GitHub repository metadata and README.md
Documents DiT-S/B/L action modules, 16-step 7-DoF action chunks, ~30 GB FP32 memory, 181 ms A6000 BF16 inference (~5.5 Hz Adaptive Action Ensemble), FSDP full fine-tuning, and MIT licensing.Revision, date & where cited
Microsoft (GitHub) · github.com · revision
b174a1b· GitHub API & README.md at commit b174a1b86dee (2025-10-30) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 7 more - S002 microsoft/CogACT root LICENSE at pinned commit
MIT License (Copyright Microsoft Corporation).Revision, date & where cited
Microsoft Corporation · github.com · revision
b174a1b· LICENSE at commit b174a1b86dee · Accessed Nov 21, 2026
Used for: Separate license matrix - S003 Hugging Face model cards and API metadata for CogACT-Small, CogACT-Base, and CogACT-Large
All three checkpoints are ungated, license: mit, single FP32 .pt files (28.14 GB Small, 28.43 GB Base, 29.24 GB Large) using DINOv2 ViT-L/14 + SigLIP ViT-So400M/14 + Llama-2 + DiT action head.Revision, date & where cited
CogACT (Hugging Face) · huggingface.co · HF API & README.md for CogACT-Small (ca302a300542), CogACT-Base (6550bf0992f1), CogACT-Large (0a7e03ce8a3c), lastModified 2024-12-04 · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 6 more
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from microsoft/CogACT@b174a1b86dee and HF CogACT-Small/Base/Large checkpoints.