RoboFlamingo
Robotics modelOpenFlamingo-based Vision-Language-Action manipulation family (3B, 4B, and 9B variants) with an LSTM/GPT/MLP policy head fine-tuned on CALVIN, publishing MIT-licensed code and six model-card-listed default PyTorch checkpoints on Hugging Face. [1] [2] [3]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Robotics and VLA researchers; CALVIN benchmark practitioners; Single-server imitation learning teams [1]
- Embodiments
- Franka Emika Panda (CALVIN simulation benchmark) [1]
- Source rights
- Named licenses are documented by part. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Overview
RoboFlamingo adapts pretrained OpenFlamingo vision-language models (3B, 4B, and 9B variants) into language-conditioned manipulation policies by pairing the VLM backbone with an explicit sequential policy head and fine-tuning on offline imitation trajectories. Both the PyTorch training/evaluation repository (RoboFlamingo/RoboFlamingo at commit 1475a07d043a) and the Hugging Face checkpoint repository (linked in the README as roboflamingo/RoboFlamingo, which the Hugging Face API resolves to robovlms/RoboFlamingo at revision b08c0a2ac4d2) declare MIT licensing. [1] [2] [3] [4]
Model card summary
- Architecture
OpenFlamingo vision-language backbone (OpenCLIP ViT-L/14 vision encoder + Perceiver Resampler + gated cross-attention into MPT-1B, RedPajama-3B, or MPT-7B LLM) paired with a sequential LSTM, GPT, or FC policy head (default post-fusion LSTM with window_size 12).
- Base model
OpenFlamingo (3B-vitl-mpt1b, 3B-vitl-mpt1b-langinstruct, 4B-vitl-rpj3b, 4B-vitl-rpj3b-langinstruct, 9B-vitl-mpt7b)
- Parameter count
3B, 4B, and 9B VLM backbone variants (plus lightweight LSTM/MLP policy head)
Exact total parameter count including policy head is not published as a single scalar; Hugging Face releases 3B, 4B, and 9B checkpoints.
- Input modalities
- rgblanguage
Static workspace RGB camera plus optional gripper/wrist RGB camera (--use_gripper) and free-form English language instruction.
- Output modalities
- ee-pose-deltagripper-command
7-DoF CALVIN continuous end-effector delta pose (6-DoF) plus binary gripper open/close action predicted per step.
- Action space
7-DoF CALVIN manipulation action space (6-DoF continuous end-effector delta pose + gripper open/close command) predicted sequentially via an LSTM/GPT/MLP decoder over a history window (default window_size 12).
- Trained / supported embodiments
- Franka Emika Panda (CALVIN simulation benchmark)
- Training datasets
- CALVIN (ABCD and ABC language-annotated splits)
- COCO Captioning and VQAv2 (optional co-fine-tuning variant)
- Interfaces
- PyTorch Python API (create_model_and_transforms)
- torchrun DDP CLI (train_calvin.py)
- CALVIN evaluation runner (eval_ckpts.py)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| MPT-IFT-3B (ABCD->D, best default) | pytorch | 17.6 GB | open | b08c0a2ac4d2 |
| MPT-3B (ABCD->D) | pytorch | 17.6 GB | open | b08c0a2ac4d2 |
| MPT-IFT-3B + COCO/VQA Co-Finetune (ABCD->D) | pytorch | 17.89 GB | open | b08c0a2ac4d2 |
| MPT-4B / RedPajama-3B Base (ABCD->D) | pytorch | 23.67 GB | open | b08c0a2ac4d2 |
| MPT-IFT-4B / RedPajama-3B Instruct (ABCD->D) | pytorch | 23.67 GB | open | b08c0a2ac4d2 |
| MPT-9B (ABCD->D) | pytorch | 42.98 GB | open | b08c0a2ac4d2 |
Six model-card-listed default .pth checkpoints verified via Hugging Face API at revision b08c0a2ac4d2 (the API resolves roboflamingo/RoboFlamingo to canonical id robovlms/RoboFlamingo and also lists three additional .pth files under ABC_D/ not detailed in the model card).
Is this for you?
- Use cases
- Evaluating language-conditioned multi-step manipulation on the CALVIN benchmark (ABCD->D, ABC->D, enriched language)
- Fine-tuning OpenFlamingo 3B/4B/9B VLMs on offline robot imitation datasets
- Co-fine-tuning robot manipulation data with COCO captioning and VQAv2 to retain general vision-language capabilities
- Known limitations
- Released checkpoints are trained specifically on the simulated CALVIN ABCD->D split (no real-robot weights are included in the Hugging Face repo).
- Checkpoints are large FP32/optimizer-inclusive PyTorch .pth files (17.6 GB to 43.0 GB each) and depend on downloading matching OpenFlamingo/LLM/tokenizer weights.
- README notes that the diffusion decoder_type option has dataloader bugs; repository targets Python 3.8 and has had no commits since April 2024.
- Safety and limitations
- Released checkpoints are trained only in the simulated CALVIN tabletop environment and require adaptation before real-robot deployment.
- Downstream commercial use must also verify the licenses of the separately downloaded OpenFlamingo and underlying LLM backbones (e.g., MPT-1B-RedPajama-200B-Dolly, RedPajama-INCITE-3B, MPT-7B).
- Default PyTorch .pth checkpoints are 17.6-43.0 GB each and rely on legacy Python 3.8 dependencies.
Compute & fine-tuning
- Inference compute
- Documented hardware: 8x NVIDIA A100 (80 GB) server used for experiments
- Precision:
fp32 - Edge / consumer capable: No
Default launch command uses --precision fp32 on 8x A100 80 GB; exact minimum single-GPU inference VRAM and latency_ms are not specified.
[1] · Editorial assessment
- Fine-tuning
- Supported: Yes
- Methods: DDP fine-tuning on CALVIN (train_calvin.py); Co-fine-tuning with COCO + VQAv2 + CALVIN; Optional frozen embeddings (freeze-emb)
- Recipe: Upstream fine-tuning documentation
- Host computer
All paper experiments were run on a single GPU server with 8x NVIDIA A100 (80 GB) GPUs; README states RoboFlamingo can be trained or evaluated on a single GPU server depending on model size.
- Operating system
- Linux
Bash launch scripts, torchrun DDP, and CALVIN simulator target Linux.
[1] · Editorial assessment
- Dependencies
- Python 3.8
- PyTorch
- OpenCLIP (ViT-L-14)
- Hugging Face Transformers
- OpenFlamingo pretrained weights
- CALVIN benchmark simulator and dataset
- Skills
- Python 3.8
- PyTorch DDP (torchrun)
- CALVIN simulation environment setup
Inferred from documented conda, torchrun, and CALVIN prerequisite instructions.
[1] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [1]
- Download the CALVIN dataset split (D, ABC, ABCD, or debug) and matching OpenFlamingo pretrained VLM/LLM weights.
- Clone RoboFlamingo/RoboFlamingo, create a Python 3.8 conda environment, and pip install -r requirements.txt.
- Configure local LLM and tokenizer paths in robot_flamingo/models/factory.py.
- Run torchrun DDP training (robot_flamingo/train/train_calvin.py) or download a checkpoint from huggingface.co/roboflamingo/RoboFlamingo and run python eval_ckpts.py.
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| CALVIN ABCD->D (language-annotated) | Avg Len 4.09 (SR 1-5: 96.4% / 89.6% / 82.4% / 74.0% / 66.2%) | Upstream-reported simulation score for MPT-3B-IFT fine-tuned checkpoint; not independently reproduced by OSR. |
| CALVIN ABC->D (zero-shot unseen environment) | Avg Len 2.48 (SR 1-5: 82.4% / 61.9% / 46.6% / 33.1% / 23.5%) | Upstream-reported simulation score; not independently reproduced by OSR. |
| CALVIN ABCD->D (Enriched language instructions) | Avg Len 2.12 with freeze-emb (1.85 standard fine-tune) | Upstream-reported simulation score; not independently reproduced by OSR. |
Project notes
Architecture and checkpoint family
RoboFlamingo combines an OpenCLIP ViT-L/14 vision encoder, Perceiver Resampler, and gated cross-attention into an underlying language model (MPT-1B-RedPajama, MPT-1B-RedPajama-Dolly, RedPajama-INCITE-3B base/instruct, or MPT-7B) with a policy decoder supporting lstm (default post-fusion LSTM with window_size: 12), GPT, fc, or diffusion (the README notes dataloader bugs in the diffusion option). At each step, the policy takes static workspace RGB and optional gripper-camera RGB (--use_gripper) alongside a free-form English instruction to output 7-DoF CALVIN actions (6-DoF end-effector delta pose plus gripper open/close). [1]
The Hugging Face model card lists six default ABCD->D .pth checkpoints: MPT-3B (..._mpt_3b_4.pth, 17.60 GB), MPT-IFT-3B (..._mpt_dolly_3b_2.pth, 17.60 GB, corresponding to the README's default best-result configuration), MPT-IFT-3B co-fine-tuned with COCO and VQAv2 (cofinetune_..._mpt_dolly_3b_3.pth, 17.89 GB), MPT-4B (..._mpt_base_4b_3.pth, 23.67 GB), MPT-IFT-4B (..._mpt_4b_3.pth, 23.67 GB), and MPT-9B (..._mpt_9b_2.pth, 42.98 GB). The repository tree also contains three additional .pth files under ABC_D/ that are not detailed in the model card. [1] [3]
Compute requirements and upstream-reported benchmarks
All paper experiments reportedly used a single server with 8x NVIDIA A100 (80 GB) GPUs, and the documented training command uses torchrun DDP with FP32 precision; the README notes that training or evaluation memory depends on model size, while exact minimum single-GPU inference VRAM and control latency are not documented. On the simulated CALVIN benchmark, the authors report an average sequence length of 4.09 on ABCD->D (SR 1–5: 96.4% / 89.6% / 82.4% / 74.0% / 66.2%, associated with MPT-3B-IFT / MPT-IFT-3B) and 2.48 on zero-shot unseen ABC->D (82.4% / 61.9% / 46.6% / 33.1% / 23.5%), plus 2.12 on enriched language instructions when freezing embeddings (freeze-emb). These scores are strictly upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or CALVIN simulation rollouts for this profile. [1]
What we checked
None. Prepared from public upstream sources without hands-on training, fine-tuning, or simulation execution.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Upstream last updated
- Apr 17, 2024 [1]
- Latest release
- Apr 12, 2024 [3]
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- RoboFlamingo/RoboFlamingo@1475a07d043a62bb07ab11abfc71115bb72e2a16; HF roboflamingo/RoboFlamingo@b08c0a2ac4d2a92927ddc066e3eb077319bc5a94 · combined hardware and software repository
Separate license matrix
Open-weights VLA manipulation family with MIT-licensed training/evaluation code and six ungated PyTorch checkpoints (3B, 4B, and 9B variants) on Hugging Face trained on the public MIT-licensed CALVIN benchmark.
| Layer | Availability / license |
|---|---|
| Code | available · MIT Verified MIT License in root LICENSE file. [1] [2] |
| Model weights | available · MIT (model card YAML; upstream OpenFlamingo/LLM backbone licenses also apply) Six ungated .pth checkpoints declared under license: mit on Hugging Face. [3] |
| Training datasets | available · MIT (CALVIN); separate upstream terms for COCO and VQAv2 Trains on public CALVIN splits (and optionally COCO/VQAv2 for co-fine-tuning). [1] |
| Documentation | available · MIT [1] [2] |
Licensing and practical caveats
While RoboFlamingo's own code and Hugging Face checkpoint repository declare MIT licensing and the CALVIN benchmark dataset is MIT-licensed, users must separately download the base OpenFlamingo and LLM/tokenizer weights and verify each upstream backbone's license terms before commercial deployment. The repository has been dormant since April–May 2024, targets Python 3.8, and publishes simulated CALVIN checkpoints only rather than real-robot weights; readers should also note the roboflamingo/RoboFlamingo to robovlms/RoboFlamingo Hugging Face namespace redirect when pinning downloads. [1] [2] [3]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 RoboFlamingo/RoboFlamingo GitHub repository README.md
Documents OpenFlamingo 3B/4B/9B VLM backbones, LSTM/GPT/FC policy heads, 8x A100 80 GB DDP training, CALVIN ABCD->D (4.09), ABC->D (2.48), and enriched (2.12) benchmark tables, and COCO/VQA co-fine-tuning.Revision, date & where cited
RoboFlamingo (GitHub) · github.com · revision
1475a07· README.md at commit 1475a07d043a (2024-04-17) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Compute & fine-tuning and 6 more - S002 RoboFlamingo/RoboFlamingo root LICENSE file
MIT License, Copyright (c) 2023 XinghangLi.Revision, date & where cited
RoboFlamingo (GitHub) · github.com · revision
1475a07· LICENSE at commit 1475a07d043a · Accessed Nov 21, 2026
Used for: Overview, Separate license matrix - S003 Hugging Face API metadata and model card: roboflamingo/RoboFlamingo
Six ungated .pth checkpoints (17.60 GB to 42.98 GB; 143.40 GB total) for 3B, 4B, 9B, and co-finetuned 3B models on CALVIN ABCD->D, declared under license: mit.Revision, date & where cited
Hugging Face (roboflamingo) · huggingface.co · HF API & README.md at revision b08c0a2ac4d2 (2024-04-12) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 4 more - S004 arXiv:2311.01378 — Vision-Language Foundation Models as Effective Robot Imitators
Introduces RoboFlamingo adapting OpenFlamingo VLMs to language-conditioned robot manipulation.Revision, date & where cited
arXiv · arxiv.org · arXiv:2311.01378v3 (2023-11-02) · Accessed Nov 21, 2026
Used for: Overview, Model card summary
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from RoboFlamingo/RoboFlamingo@1475a07d043a and HF roboflamingo/RoboFlamingo@b08c0a2ac4d2.