GR-MG
Robotics modelTwo-stage multi-modal goal-conditioned manipulation policy pairing an InstructPix2Pix goal-image generator with an Ego4D-pretrained GPT-2/MAE/CLIP policy network, releasing Apache-2.0 code and three open checkpoints for CALVIN. [1] [2] [3] [4]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Robotics and visuomotor policy researchers; CALVIN benchmark practitioners; Goal-conditioned imitation learning researchers [1]
- Embodiments
- Franka Emika Panda (CALVIN simulation benchmark) [1]
- Source rights
- Named licenses are documented by part. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
Overview
GR-MG (*Leveraging Partially Annotated Data via Multi-Modal Goal-Conditioned Policy*, IEEE RA-L / arXiv:2408.14368) is a two-stage visuomotor manipulation system that decouples language-to-visual-subgoal planning from action execution so that actionless human activity videos (Ego4D) and textless robot trajectories can both contribute to policy learning. The official repository (bytedance/GR-MG at commit f45b0725e462) is licensed under Apache-2.0 and links three publicly downloadable checkpoints verified live via HTTP 200. [1] [2] [3] [4]
Model card summary
- Architecture
Two-component system: (1) InstructPix2Pix-based goal image generation model; (2) GR_MG policy combining a ViT-Base MAE image encoder, CLIP text encoder, Perceiver Resampler (depth 3, 4 heads, 9 latents), and 12-layer / 12-head GPT-2 transformer (embed_dim 384, seq_len 10) with a CVAE-style action decoder (act_latent_dim 32) and progress decoder.
Hyperparameters verified directly from policy/config/train.json and policy/model/model.py.
- Base model
InstructPix2Pix (timbrooks/instruct-pix2pix) + MAE ViT-Base + CLIP + GR-1 Ego4D pretraining recipe
- Parameter count
Not stated as a single scalar in README (policy checkpoint is 1.28 GB; Ego4D pretrained checkpoint is 1.24 GB; InstructPix2Pix goal generator checkpoint is 13.75 GB including optimizer/EMA state)
Exact parameter counts are not listed in README.md; checkpoint sizes verified via HTTP HEAD.
- Input modalities
- rgbproprioceptionlanguage
Workspace RGB (and optional hand RGB), language instruction, generated goal RGB image, and optional proprioceptive state (use_state flag in GR_MG).
- Output modalities
- ee-pose-deltagripper-commandvideo-frames
Goal generator outputs predicted goal RGB image frames; policy predicts 5-step CALVIN action chunks (act_len: 5) plus progress predictions.
- Action space
Continuous 5-step action chunks (act_len: 5, act_latent_dim: 32) over a 10-step observation history (seq_len: 10) for 7-DoF CALVIN manipulation (end-effector delta pose + gripper command).
Verified from policy/config/train.json.
- Trained / supported embodiments
- Franka Emika Panda (CALVIN simulation benchmark)
- Training datasets
- Ego4D (human activity videos for policy pretraining)
- CALVIN (robot manipulation trajectories)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| GR-MG Multi-modal Goal Conditioned Policy (epoch=47-step=83712.ckpt) | pytorch | 1.28 GB | open | etag:1e47858a7734079f920401ed4391eca0-24 |
| GR-MG Goal Image Generation Model (goal_gen.ckpt) | pytorch | 13.75 GB | open | etag:346840e04b852ddccb5b9802cda0090e-262 |
| GR-MG Ego4D Pretrained Policy (pretrained.pt) | pytorch | 1.24 GB | open | etag:5a3ccfd28f74ff03c7fd6f47587d1754-23 |
All three direct download URLs verified HTTP 200 with exact sizes (1.281 GB, 13.754 GB, 1.244 GB) and ETags on 2026-11-21.
Is this for you?
- Use cases
- Evaluating multi-modal goal-conditioned manipulation on the CALVIN benchmark
- Fine-tuning a goal-image diffusion model (InstructPix2Pix) for visual subgoal generation
- Pretraining a GPT-2-based visuomotor policy on Ego4D human activity videos before robot fine-tuning
- Known limitations
- Tested upstream only with CUDA 12.1 and Python 3.9.
- Goal image generation checkpoint (goal_gen.ckpt, 13.75 GB) derives from InstructPix2Pix/Stable Diffusion and requires downloading upstream timbrooks/instruct-pix2pix and MAE ViT-Base weights.
- Reproducing Stage-2 Ego4D pretraining from scratch requires separately obtaining access to the Ego4D dataset; released checkpoints target CALVIN simulation.
- Safety and limitations
- Checkpoints are hosted as direct object-storage downloads without a separate per-file weight license; downstream use must also respect upstream InstructPix2Pix/Stable Diffusion, MAE, CLIP, and Ego4D terms.
- Released checkpoints are trained for the simulated CALVIN benchmark and are not directly validated on arbitrary physical robots.
- Repository is tested only on CUDA 12.1 and Python 3.9 and has had no commits since January 2025.
Compute & fine-tuning
- Inference compute
- Documented hardware: CUDA 12.1 GPU(s) (exact GPU model not stated in README)
- Precision:
bf16 - Edge / consumer capable: No
Both train.json configs set precision: bf16; exact minimum VRAM and inference latency_ms are not documented.
[1] · Editorial assessment
- Fine-tuning
- Supported: Yes
- Methods: Goal image generator training (goal_gen/train_ip2p.sh); Ego4D policy pretraining (policy/main.sh policy/config/pretrain.json); Policy fine-tuning on CALVIN (policy/main.sh policy/config/train.json)
- Recipe: Upstream fine-tuning documentation
- Host computer
Multi-GPU CUDA 12.1 DDP training configured with BF16 precision (batch_size 64 for goal_gen, 32 for policy); multi-GPU CALVIN evaluation is supported.
- Operating system
- Linux
CUDA 12.1, bash install scripts, and CALVIN target Linux.
[1] · Editorial assessment
- Dependencies
- Python 3.9
- CUDA 12.1
- PyTorch & PyTorch Lightning
- InstructPix2Pix (timbrooks/instruct-pix2pix)
- MAE ViT-Base (mae_pretrain_vit_base.pth)
- CLIP
- CALVIN simulator and dataset
- Skills
- Python 3.9
- PyTorch / PyTorch Lightning DDP
- CALVIN benchmark setup
Inferred from documented PyTorch Lightning DDP configs and shell scripts.
[1] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [1]
- Clone bytedance/GR-MG on a CUDA 12.1 / Python 3.9 Linux environment and run bash ./goal_gen/install.sh and bash ./policy/install.sh.
- Download pretrained InstructPix2Pix (timbrooks/instruct-pix2pix) into resources/IP2P/ and MAE ViT-Base (mae_pretrain_vit_base.pth) into resources/MAE/.
- Download and unzip the CALVIN dataset and fetch epoch=47-step=83712.ckpt and goal_gen.ckpt (plus pretrained.pt for training).
- Run bash ./evaluate/eval.sh ./policy/config/train.json for CALVIN evaluation or bash ./policy/main.sh ./policy/config/train.json for policy training.
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| GR-MG paper evaluations (IEEE RA-L / arXiv:2408.14368; CALVIN and real-robot tasks) | See arXiv:2408.14368 (no numeric table in repository README) | Upstream-reported paper evaluations; not independently verified or reproduced by OSR. |
Project notes
Two-stage architecture and verified checkpoints
In policy/model/model.py and policy/config/train.json, GR-MG pairs an InstructPix2Pix goal-image generator (goal_gen.ckpt, 13.75 GB) with a multi-modal goal-conditioned policy (epoch=47-step=83712.ckpt, 1.28 GB) initialized from an Ego4D-pretrained checkpoint (pretrained.pt, 1.24 GB). The policy combines a ViT-Base MAE image encoder, CLIP language encoder, Perceiver Resampler (depth: 3, 4 heads, 9 latents), and a 12-layer / 12-head GPT-2 transformer (embed_dim: 384, seq_len: 10), predicting 5-step continuous action chunks (act_len: 5) for CALVIN manipulation through its CVAE-style action decoder (act_latent_dim: 32) while separately predicting task progress through its progress decoder. Exact total parameter counts are not reported in the README. [1] [3]
Setup, compute, and upstream-reported evaluations
The authors note that GR-MG is tested only with CUDA 12.1 and Python 3.9, using PyTorch Lightning DDP with precision: bf16 (batch size 64 for goal_gen, 32 for policy). Setup requires downloading upstream timbrooks/instruct-pix2pix weights, mae_pretrain_vit_base.pth, and the CALVIN dataset, and pretraining from scratch additionally requires obtaining Ego4D access. As an upstream evaluation workflow, the README documents running bash ./evaluate/eval.sh ./policy/config/train.json, with the goal generator and policy checkpoints selected inside eval.sh. The released checkpoints target CALVIN simulation, and the supplied evidence does not establish direct applicability to arbitrary physical robots. Exact minimum GPU VRAM, inference latency, and numeric CALVIN benchmark tables are not included in the repository README (benchmark results are reported in arXiv:2408.14368); Open Source Robotics has not run training, fine-tuning, inference, or CALVIN evaluations for this profile. [1] [4]
What we checked
None. Prepared from public upstream sources without hands-on training, fine-tuning, or simulation execution.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
Separate license matrix
Apache-2.0 official implementation of an IEEE RA-L manipulation policy with three publicly downloadable checkpoints (CALVIN policy, InstructPix2Pix goal generator, and Ego4D pretrained policy) verified live.
| Layer | Availability / license |
|---|---|
| Code | available · Apache-2.0 Verified Apache License Version 2.0 in root LICENSE file. [1] [2] |
| Model weights | available · No separate weight license file (repo is Apache-2.0; upstream InstructPix2Pix/Stable Diffusion, MAE, CLIP, and Ego4D terms apply) All three ByteTOS checkpoint downloads (policy, goal_gen, pretrained) are publicly accessible without authentication. [1] [2] [3] |
| Training datasets | available · MIT (CALVIN); Ego4D requires separate Ego4D data license agreement CALVIN is openly downloadable under MIT; Ego4D requires registration/license. [1] |
| Documentation | available · Apache-2.0 [1] [2] |
Licensing boundaries
The bytedance/GR-MG codebase is Apache-2.0, and all three ByteTOS checkpoints are directly downloadable without gating. However, the checkpoints do not include a separate per-file weight license, and downstream deployment must account for upstream licenses on InstructPix2Pix (built on Stable Diffusion), MAE ViT-Base, CLIP, CALVIN (MIT), and Ego4D. [1] [2] [3]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 bytedance/GR-MG GitHub repository README.md, policy/config/train.json, goal_gen/config/train.json, and policy/model/model.py
Documents two-stage InstructPix2Pix goal generator + GPT-2/MAE/CLIP/Perceiver Resampler policy (12 layers, 12 heads, embed_dim 384, seq_len 10, act_len 5, BF16 DDP), CUDA 12.1 / Python 3.9 requirement, and three checkpoint URLs.Revision, date & where cited
bytedance (GitHub) · github.com · revision
f45b072· README.md and configs at commit f45b0725e462 (2025-01-12) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 7 more - S002 bytedance/GR-MG root LICENSE file
Revision, date & where cited
bytedance (GitHub) · github.com · revision
f45b072· LICENSE at commit f45b0725e462 · Accessed Nov 21, 2026
Used for: Overview, Is this for you?, Separate license matrix - S003 HTTP HEAD verification of GR-MG ByteTOS checkpoint artifacts
All three URLs return HTTP 200 (Last-Modified 2024-08-26): epoch=47-step=83712.ckpt (1.281 GB), goal_gen.ckpt (13.754 GB), pretrained.pt (1.244 GB).Revision, date & where cited
ByteDance (lf-robot-opensource.bytetos.com) · lf-robot-opensource.bytetos.com · HTTP HEAD on epoch=47-step=83712.ckpt, goal_gen.ckpt, pretrained.pt · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 3 more - S004 arXiv:2408.14368 — GR-MG: Leveraging Partially Annotated Data via Multi-Modal Goal-Conditioned Policy
Introduces GR-MG leveraging human videos without action labels and robot trajectories without text labels via multi-modal goal conditioning.Revision, date & where cited
arXiv / IEEE RA-L · arxiv.org · arXiv:2408.14368v2 (2024-08-26) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Upstream-reported benchmarks, Setup, compute, and upstream-reported evaluations
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from bytedance/GR-MG@f45b0725e462 and HTTP-verified ByteTOS checkpoints.