Open Source RoboticsRobots, software & learning

CogACT

Robotics model

Componentized Vision-Language-Action model family from Microsoft Research Asia pairing a 7B Prismatic VLM (DINOv2 + SigLIP + Llama-2) with specialized Diffusion Transformer action modules (DiT-Small, DiT-Base, DiT-Large) and MIT-licensed code and weights. [1] [2] [3]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Robotics researchers studying diffusion-head VLA architectures; Manipulation practitioners evaluating SimplerEnv or fine-tuning on RLDS datasets [1] [3]
Parameters
~7.6B–7.9B total (7B Prismatic VLM backbone + DiT-S / DiT-B / DiT-L action head [1] [3]
Embodiments
Google Robot (SimplerEnv Fractal / fractal20220817_data), WidowX (SimplerEnv Bridge / BridgeData V2), Open… [1] [3]
Source rights
Named licenses are documented by part. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware

Requirements · Research details and dates

On this page

Overview

CogACT (microsoft/CogACT, arXiv:2411.19650) is a Vision-Language-Action model family from Microsoft Research Asia that decouples high-level visual-language cognition from continuous robot action synthesis. Instead of quantizing actions directly into VLM tokens, CogACT extracts a single cognition token from a 7B Prismatic VLM backbone (DINOv2 ViT-L/14 + SigLIP ViT-So400M/14 + Llama-2) and conditions a specialized Diffusion Transformer (DiT-S, DiT-B, or DiT-L) action head trained on an Open X-Embodiment (oxe_magic_soup_plus_minus) RLDS mixture. [1] [3]

Model card summary

Architecture

Componentized VLA combining a Prismatic VLM backbone (DINOv2 ViT-L/14 + SigLIP ViT-So400M/14 vision encoders and Llama-2 7B) emitting a single cognition token to condition a Diffusion Transformer action module (DiT-S, DiT-B, or DiT-L) sampled via DDIM (10 steps, CFG).

[1] [3]

Base model

openvla/openvla-7b-prismatic (Prismatic VLM: DINOv2 ViT-L/14 + SigLIP ViT-So400M/14 + Llama-2 7B)

README documents initializing from openvla/openvla-7b-prismatic or PrismaticVLM.

[1] [3]

Parameter count

~7.6B–7.9B total (7B Prismatic VLM backbone + DiT-S / DiT-B / DiT-L action head; exact scalar param count not exposed in .pt HF metadata; FP32 checkpoints are 28.14 GB, 28.43 GB, and 29.24 GB)

Checkpoints are raw PyTorch .pt files without HF safetensors parameter-count metadata; size scaling across Small (28.14 GB), Base (28.43 GB), and Large (29.24 GB) reflects the DiT action module.

[1] [3] · Editorial assessment

Input modalities
rgblanguage

Single-view RGB camera image and English language instruction.

[1] [3]

Output modalities
ee-pose-deltagripper-command

7-DoF end-effector deltas (x, y, z, roll, pitch, yaw, gripper open/close).

[1] [3]

Action space

Continuous 16-step action chunks of 7-DoF normalized end-effector deltas (shape [16, 7], future_action_window_size=15), sampled with 10-step DDIM and classifier-free guidance (cfg_scale 1.5–7), optionally combined via Adaptive Action Ensemble.

[1] [3]

Control frequency

~5.5 Hz on a single NVIDIA A6000 in bfloat16 with Adaptive Action Ensemble (181 ms per 16-action inference; up to k times higher with k-step open-loop chunk execution)

[1]

Trained / supported embodiments
  • Google Robot (SimplerEnv Fractal / fractal20220817_data)
  • WidowX (SimplerEnv Bridge / BridgeData V2)
  • Open X-Embodiment 7-DoF EEF-delta manipulators
  • Custom 7-DoF manipulators via RLDS fine-tuning

[1] [3]

Training datasets
  • Open X-Embodiment subset (oxe_magic_soup_plus_minus RLDS mixture)

[1] [3]

Interfaces
  • Python load_vla / predict_action / predict_action_batch API
  • HTTP REST inference server (scripts/deploy.py, /api/inference)
  • SimplerEnv CogACTInference policy wrapper
  • torchrun FSDP training CLI (scripts/train.py)

[1]

Checkpoints

CheckpointFormatSizeAccessPinned revision
CogACT/CogACT-Smallpytorch28.14 GBopenca302a300542
CogACT/CogACT-Basepytorch28.43 GBopen6550bf0992f1
CogACT/CogACT-Largepytorch29.24 GBopen0a7e03ce8a3c

Three ungated FP32 PyTorch .pt checkpoints verified via Hugging Face API.

[1] [3]

Is this for you?

Use cases
  • Zero-shot single-camera RGB manipulation inference on Open X-Embodiment setups (e.g., Google Robot Fractal, WidowX Bridge)
  • FSDP full fine-tuning on custom RLDS manipulation datasets
  • HTTP server-client real-robot deployment with Adaptive Action Ensemble

[1] [3]

Known limitations
  • Published Hugging Face checkpoints are single FP32 PyTorch .pt files (~28.1–29.2 GB each) requiring ~30 GB GPU memory unless VLM weights are cast to bfloat16 after loading
  • Single-view RGB input only (no proprioceptive state or multi-camera fusion in the released checkpoints)
  • Upstream recommends multi-GPU FSDP full fine-tuning (8x A100 example) rather than LoRA
  • Increasing open-loop action chunk execution steps k (up to 16) trades accuracy for higher control frequency

[1] [3]

Safety and limitations
  • Checkpoints are distributed as raw PyTorch .pt pickle files (~28–29 GB); users should verify checkpoint integrity and trust before deserializing.
  • Single-view RGB policy without proprioception or collision/safety guardrails; real-world deployment requires dataset-specific fine-tuning and robot-side safety limits.
  • Built on Prismatic / Llama-2 7B backbone provenance alongside MIT-licensed CogACT code and checkpoints.

[1] [3]

Compute & fine-tuning

Inference compute
  • Minimum VRAM: 30 GB
  • Documented hardware: NVIDIA RTX A6000 (bfloat16)
  • Precision: fp32 / bf16
  • Reported latency: 181 ms
  • Edge / consumer capable: No

README notes ~30 GB memory in FP32 with optional bfloat16 VLM casting; 181 ms average latency on A6000 in bfloat16 (vs 307 ms for single-step OpenVLA on the same device).

[1] [3]

Fine-tuning
  • Supported: Yes
  • Methods: PyTorch FSDP full fine-tuning via scripts/train.py (~30 epochs recommended); Pretraining from openvla/openvla-7b-prismatic or PrismaticVLM
  • Recipe: Upstream fine-tuning documentation

Upstream explicitly recommends full FSDP fine-tuning over LoRA and provides an 8x A100 command; exact minimum single-GPU VRAM is not stated.

[1]

Host computer

Inference requires ~30 GB GPU memory in FP32 (optional bfloat16 VLM casting supported; benchmarked on a single NVIDIA A6000); full FSDP fine-tuning example uses 1 node with 8x A100 GPUs.

[1] [3]

Operating system
  • Linux

Standard Linux conda, CUDA 12, flash-attn, and FSDP workflow documented.

[1] · Editorial assessment

Dependencies
  • Python >= 3.8 (3.10 recommended)
  • PyTorch >= 2.2.0 and CUDA >= 12.0
  • flash-attn==2.5.5 (optional, required for training)
  • SimplerEnv (for simulation evaluation)
  • requests (for standalone HTTP deployment client)

[1]

Skills
  • Python 3.10 and PyTorch / CUDA
  • PyTorch FSDP multi-GPU training and RLDS dataset conversion
  • HTTP requests client integration for robot control

Inferred from documented conda, FSDP, RLDS, and HTTP deployment instructions.

[1] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1] · Model card / documentation [1]

  • Create a Python 3.10 conda environment with PyTorch >= 2.2.0 and CUDA >= 12.0
  • Clone microsoft/CogACT and run pip install -e . (or pip install -e .[train] plus flash-attn==2.5.5 for training)
  • Load CogACT/CogACT-Small, CogACT-Base, or CogACT-Large via load_vla with matching action_model_type (DiT-S, DiT-B, or DiT-L)
  • Run model.predict_action with DDIM sampling (10 steps, cfg_scale 1.5) or serve via scripts/deploy.py on port 5500

[1] [3]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
CogACT-Base inference latency vs OpenVLA (NVIDIA A6000, bfloat16)181 ms for 16 generated actions (~5.5 Hz with Adaptive Action Ensemble) vs 307 ms for 1 action on OpenVLAUpstream-reported timing averaged over 100 calls on a single A6000; full SIMPLER and real-robot success tables are in arXiv:2411.19650 and not independently verified by OSR.

[1]

Project notes

Checkpoint family and 16-step DDIM action chunks

Microsoft Research Asia publishes three ungated FP32 PyTorch .pt checkpoints on Hugging Face: CogACT/CogACT-Small (28.14 GB, DiT-S), CogACT/CogACT-Base (28.43 GB, DiT-B), and CogACT/CogACT-Large (29.24 GB, DiT-L). Each takes a single-view RGB image and an English instruction (no proprioceptive state input) and generates a 16-step chunk of 7-DoF normalized end-effector deltas (x, y, z, roll, pitch, yaw, gripper) using 10-step DDIM sampling with classifier-free guidance (cfg_scale 1.5–7). Predicted chunks are unnormalized per dataset statistics (unnorm_key, e.g., fractal20220817_data) and optionally blended across steps with the repository's Adaptive Action Ensemble. Because checkpoints are raw .pt files rather than safetensors, exact scalar parameter counts per DiT variant are not exposed in Hugging Face API metadata, and users should verify file integrity before deserializing. [1] [3]

Compute footprint, deployment server, and upstream timing

Loading a CogACT checkpoint in FP32 uses ~30 GB GPU memory, though model.vlm = model.vlm.to(torch.bfloat16) reduces VLM memory footprint. On a single NVIDIA A6000 in bfloat16, upstream reports 181 ms average inference latency for CogACT-Base (~5.5 Hz with Adaptive Action Ensemble, generating 16 actions per call) compared with 307 ms for single-step autoregressive OpenVLA on the same device; executing k steps open-loop scales effective rate up to k× at the cost of accuracy over longer horizons. For real-robot deployment, scripts/deploy.py exposes an HTTP /api/inference endpoint usable from a minimal Python requests client, and sim_cogact/ integrates with SimplerEnv. For adaptation, upstream explicitly recommends PyTorch FSDP full fine-tuning (~30 epochs; 8× A100 command documented) over LoRA, and does not specify a minimum single-GPU fine-tuning VRAM. All latency and benchmark figures are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [3]

What we checked

None. Prepared from public upstream sources and Hugging Face API metadata without hands-on training, fine-tuning, or inference.

Updates

Upstream last updated
Oct 30, 2025 [1]
Latest release
Dec 4, 2024 [3]
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
microsoft/CogACT@b174a1b86deedfab4d198d935207e7bb0527994e; HF CogACT/CogACT-Base@6550bf0992f162fc5d74f14ffee30771a9433363 · combined hardware and software repository

Separate license matrix

MIT-licensed diffusion-head VLA family from Microsoft Research Asia with three ungated Hugging Face checkpoints (Small, Base, Large), FSDP training code, and public Open X-Embodiment training recipe.

LayerAvailability / license
Codeavailable · MIT
Verified MIT License (Microsoft Corporation) in root LICENSE file. [1] [2]
Model weightsavailable · MIT
README and all three HF model cards declare MIT license for code and model weights. [1] [3]
Training datasetsavailable · Mixed (Open X-Embodiment RLDS mixture; per-dataset upstream licenses apply)
[1] [3]
Documentationavailable · MIT
[1] [2]

MIT licensing and upstream dataset provenance

The microsoft/CogACT repository (verified in the root LICENSE file) and all three Hugging Face model cards declare the MIT License for code and model weights. Under the supplied MIT License text, users may use, copy, modify, merge, publish, distribute, sublicense, and sell copies provided the copyright and permission notice are retained in all copies or substantial portions, with the software supplied "as is" and without warranty. Pretraining uses a subset of the public Open X-Embodiment RLDS corpus (oxe_magic_soup_plus_minus), where individual constituent datasets retain their own upstream licenses, and training-from-scratch recipes initialize from openvla/openvla-7b-prismatic (a Llama-2 7B-based backbone). This review verifies CogACT's MIT declarations but does not resolve whether underlying base-model license obligations also apply; obtain and review the applicable base-model license texts before finalizing compliance for redistribution or commercial deployment. [1] [2] [3]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from microsoft/CogACT@b174a1b86dee and HF CogACT-Small/Base/Large checkpoints.