Open Source RoboticsRobots, software & learning

RoboFlamingo

Robotics model

OpenFlamingo-based Vision-Language-Action manipulation family (3B, 4B, and 9B variants) with an LSTM/GPT/MLP policy head fine-tuned on CALVIN, publishing MIT-licensed code and six model-card-listed default PyTorch checkpoints on Hugging Face. [1] [2] [3]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Robotics and VLA researchers; CALVIN benchmark practitioners; Single-server imitation learning teams [1]
Parameters
3B, 4B, and 9B VLM backbone variants (plus lightweight LSTM/MLP policy head) [1] [3]
Embodiments
Franka Emika Panda (CALVIN simulation benchmark) [1]
Source rights
Named licenses are documented by part. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware

Requirements · Research details and dates

On this page

Overview

RoboFlamingo adapts pretrained OpenFlamingo vision-language models (3B, 4B, and 9B variants) into language-conditioned manipulation policies by pairing the VLM backbone with an explicit sequential policy head and fine-tuning on offline imitation trajectories. Both the PyTorch training/evaluation repository (RoboFlamingo/RoboFlamingo at commit 1475a07d043a) and the Hugging Face checkpoint repository (linked in the README as roboflamingo/RoboFlamingo, which the Hugging Face API resolves to robovlms/RoboFlamingo at revision b08c0a2ac4d2) declare MIT licensing. [1] [2] [3] [4]

Model card summary

Architecture

OpenFlamingo vision-language backbone (OpenCLIP ViT-L/14 vision encoder + Perceiver Resampler + gated cross-attention into MPT-1B, RedPajama-3B, or MPT-7B LLM) paired with a sequential LSTM, GPT, or FC policy head (default post-fusion LSTM with window_size 12).

[1] [4]

Base model

OpenFlamingo (3B-vitl-mpt1b, 3B-vitl-mpt1b-langinstruct, 4B-vitl-rpj3b, 4B-vitl-rpj3b-langinstruct, 9B-vitl-mpt7b)

[1]

Parameter count

3B, 4B, and 9B VLM backbone variants (plus lightweight LSTM/MLP policy head)

Exact total parameter count including policy head is not published as a single scalar; Hugging Face releases 3B, 4B, and 9B checkpoints.

[1] [3]

Input modalities
rgblanguage

Static workspace RGB camera plus optional gripper/wrist RGB camera (--use_gripper) and free-form English language instruction.

[1]

Output modalities
ee-pose-deltagripper-command

7-DoF CALVIN continuous end-effector delta pose (6-DoF) plus binary gripper open/close action predicted per step.

[1] [4]

Action space

7-DoF CALVIN manipulation action space (6-DoF continuous end-effector delta pose + gripper open/close command) predicted sequentially via an LSTM/GPT/MLP decoder over a history window (default window_size 12).

[1] [4]

Trained / supported embodiments
  • Franka Emika Panda (CALVIN simulation benchmark)

[1]

Training datasets
  • CALVIN (ABCD and ABC language-annotated splits)
  • COCO Captioning and VQAv2 (optional co-fine-tuning variant)

[1]

Interfaces
  • PyTorch Python API (create_model_and_transforms)
  • torchrun DDP CLI (train_calvin.py)
  • CALVIN evaluation runner (eval_ckpts.py)

[1]

Checkpoints

CheckpointFormatSizeAccessPinned revision
MPT-IFT-3B (ABCD->D, best default)pytorch17.6 GBopenb08c0a2ac4d2
MPT-3B (ABCD->D)pytorch17.6 GBopenb08c0a2ac4d2
MPT-IFT-3B + COCO/VQA Co-Finetune (ABCD->D)pytorch17.89 GBopenb08c0a2ac4d2
MPT-4B / RedPajama-3B Base (ABCD->D)pytorch23.67 GBopenb08c0a2ac4d2
MPT-IFT-4B / RedPajama-3B Instruct (ABCD->D)pytorch23.67 GBopenb08c0a2ac4d2
MPT-9B (ABCD->D)pytorch42.98 GBopenb08c0a2ac4d2

Six model-card-listed default .pth checkpoints verified via Hugging Face API at revision b08c0a2ac4d2 (the API resolves roboflamingo/RoboFlamingo to canonical id robovlms/RoboFlamingo and also lists three additional .pth files under ABC_D/ not detailed in the model card).

[3]

Is this for you?

Use cases
  • Evaluating language-conditioned multi-step manipulation on the CALVIN benchmark (ABCD->D, ABC->D, enriched language)
  • Fine-tuning OpenFlamingo 3B/4B/9B VLMs on offline robot imitation datasets
  • Co-fine-tuning robot manipulation data with COCO captioning and VQAv2 to retain general vision-language capabilities

[1]

Known limitations
  • Released checkpoints are trained specifically on the simulated CALVIN ABCD->D split (no real-robot weights are included in the Hugging Face repo).
  • Checkpoints are large FP32/optimizer-inclusive PyTorch .pth files (17.6 GB to 43.0 GB each) and depend on downloading matching OpenFlamingo/LLM/tokenizer weights.
  • README notes that the diffusion decoder_type option has dataloader bugs; repository targets Python 3.8 and has had no commits since April 2024.

[1] [3]

Safety and limitations
  • Released checkpoints are trained only in the simulated CALVIN tabletop environment and require adaptation before real-robot deployment.
  • Downstream commercial use must also verify the licenses of the separately downloaded OpenFlamingo and underlying LLM backbones (e.g., MPT-1B-RedPajama-200B-Dolly, RedPajama-INCITE-3B, MPT-7B).
  • Default PyTorch .pth checkpoints are 17.6-43.0 GB each and rely on legacy Python 3.8 dependencies.

[1] [3]

Compute & fine-tuning

Inference compute
  • Documented hardware: 8x NVIDIA A100 (80 GB) server used for experiments
  • Precision: fp32
  • Edge / consumer capable: No

Default launch command uses --precision fp32 on 8x A100 80 GB; exact minimum single-GPU inference VRAM and latency_ms are not specified.

[1] · Editorial assessment

Fine-tuning
  • Supported: Yes
  • Methods: DDP fine-tuning on CALVIN (train_calvin.py); Co-fine-tuning with COCO + VQAv2 + CALVIN; Optional frozen embeddings (freeze-emb)
  • Recipe: Upstream fine-tuning documentation

[1]

Host computer

All paper experiments were run on a single GPU server with 8x NVIDIA A100 (80 GB) GPUs; README states RoboFlamingo can be trained or evaluated on a single GPU server depending on model size.

[1]

Operating system
  • Linux

Bash launch scripts, torchrun DDP, and CALVIN simulator target Linux.

[1] · Editorial assessment

Dependencies
  • Python 3.8
  • PyTorch
  • OpenCLIP (ViT-L-14)
  • Hugging Face Transformers
  • OpenFlamingo pretrained weights
  • CALVIN benchmark simulator and dataset

[1]

Skills
  • Python 3.8
  • PyTorch DDP (torchrun)
  • CALVIN simulation environment setup

Inferred from documented conda, torchrun, and CALVIN prerequisite instructions.

[1] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1] · Model card / documentation [1]

  • Download the CALVIN dataset split (D, ABC, ABCD, or debug) and matching OpenFlamingo pretrained VLM/LLM weights.
  • Clone RoboFlamingo/RoboFlamingo, create a Python 3.8 conda environment, and pip install -r requirements.txt.
  • Configure local LLM and tokenizer paths in robot_flamingo/models/factory.py.
  • Run torchrun DDP training (robot_flamingo/train/train_calvin.py) or download a checkpoint from huggingface.co/roboflamingo/RoboFlamingo and run python eval_ckpts.py.

[1] [3]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
CALVIN ABCD->D (language-annotated)Avg Len 4.09 (SR 1-5: 96.4% / 89.6% / 82.4% / 74.0% / 66.2%)Upstream-reported simulation score for MPT-3B-IFT fine-tuned checkpoint; not independently reproduced by OSR.
CALVIN ABC->D (zero-shot unseen environment)Avg Len 2.48 (SR 1-5: 82.4% / 61.9% / 46.6% / 33.1% / 23.5%)Upstream-reported simulation score; not independently reproduced by OSR.
CALVIN ABCD->D (Enriched language instructions)Avg Len 2.12 with freeze-emb (1.85 standard fine-tune)Upstream-reported simulation score; not independently reproduced by OSR.

[1]

Project notes

Architecture and checkpoint family

RoboFlamingo combines an OpenCLIP ViT-L/14 vision encoder, Perceiver Resampler, and gated cross-attention into an underlying language model (MPT-1B-RedPajama, MPT-1B-RedPajama-Dolly, RedPajama-INCITE-3B base/instruct, or MPT-7B) with a policy decoder supporting lstm (default post-fusion LSTM with window_size: 12), GPT, fc, or diffusion (the README notes dataloader bugs in the diffusion option). At each step, the policy takes static workspace RGB and optional gripper-camera RGB (--use_gripper) alongside a free-form English instruction to output 7-DoF CALVIN actions (6-DoF end-effector delta pose plus gripper open/close). [1]

The Hugging Face model card lists six default ABCD->D .pth checkpoints: MPT-3B (..._mpt_3b_4.pth, 17.60 GB), MPT-IFT-3B (..._mpt_dolly_3b_2.pth, 17.60 GB, corresponding to the README's default best-result configuration), MPT-IFT-3B co-fine-tuned with COCO and VQAv2 (cofinetune_..._mpt_dolly_3b_3.pth, 17.89 GB), MPT-4B (..._mpt_base_4b_3.pth, 23.67 GB), MPT-IFT-4B (..._mpt_4b_3.pth, 23.67 GB), and MPT-9B (..._mpt_9b_2.pth, 42.98 GB). The repository tree also contains three additional .pth files under ABC_D/ that are not detailed in the model card. [1] [3]

Compute requirements and upstream-reported benchmarks

All paper experiments reportedly used a single server with 8x NVIDIA A100 (80 GB) GPUs, and the documented training command uses torchrun DDP with FP32 precision; the README notes that training or evaluation memory depends on model size, while exact minimum single-GPU inference VRAM and control latency are not documented. On the simulated CALVIN benchmark, the authors report an average sequence length of 4.09 on ABCD->D (SR 1–5: 96.4% / 89.6% / 82.4% / 74.0% / 66.2%, associated with MPT-3B-IFT / MPT-IFT-3B) and 2.48 on zero-shot unseen ABC->D (82.4% / 61.9% / 46.6% / 33.1% / 23.5%), plus 2.12 on enriched language instructions when freezing embeddings (freeze-emb). These scores are strictly upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or CALVIN simulation rollouts for this profile. [1]

What we checked

None. Prepared from public upstream sources without hands-on training, fine-tuning, or simulation execution.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Apr 17, 2024 [1]
Latest release
Apr 12, 2024 [3]
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
RoboFlamingo/RoboFlamingo@1475a07d043a62bb07ab11abfc71115bb72e2a16; HF roboflamingo/RoboFlamingo@b08c0a2ac4d2a92927ddc066e3eb077319bc5a94 · combined hardware and software repository

Separate license matrix

Open-weights VLA manipulation family with MIT-licensed training/evaluation code and six ungated PyTorch checkpoints (3B, 4B, and 9B variants) on Hugging Face trained on the public MIT-licensed CALVIN benchmark.

LayerAvailability / license
Codeavailable · MIT
Verified MIT License in root LICENSE file. [1] [2]
Model weightsavailable · MIT (model card YAML; upstream OpenFlamingo/LLM backbone licenses also apply)
Six ungated .pth checkpoints declared under license: mit on Hugging Face. [3]
Training datasetsavailable · MIT (CALVIN); separate upstream terms for COCO and VQAv2
Trains on public CALVIN splits (and optionally COCO/VQAv2 for co-fine-tuning). [1]
Documentationavailable · MIT
[1] [2]

Licensing and practical caveats

While RoboFlamingo's own code and Hugging Face checkpoint repository declare MIT licensing and the CALVIN benchmark dataset is MIT-licensed, users must separately download the base OpenFlamingo and LLM/tokenizer weights and verify each upstream backbone's license terms before commercial deployment. The repository has been dormant since April–May 2024, targets Python 3.8, and publishes simulated CALVIN checkpoints only rather than real-robot weights; readers should also note the roboflamingo/RoboFlamingo to robovlms/RoboFlamingo Hugging Face namespace redirect when pinning downloads. [1] [2] [3]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from RoboFlamingo/RoboFlamingo@1475a07d043a and HF roboflamingo/RoboFlamingo@b08c0a2ac4d2.