Open Source RoboticsRobots, software & learning

GR-MG

Robotics model

Two-stage multi-modal goal-conditioned manipulation policy pairing an InstructPix2Pix goal-image generator with an Ego4D-pretrained GPT-2/MAE/CLIP policy network, releasing Apache-2.0 code and three open checkpoints for CALVIN. [1] [2] [3] [4]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Robotics and visuomotor policy researchers; CALVIN benchmark practitioners; Goal-conditioned imitation learning researchers [1]
Parameters
Not stated as a single scalar in README (policy checkpoint is 1.28 GB [1] [3]
Embodiments
Franka Emika Panda (CALVIN simulation benchmark) [1]
Source rights
Named licenses are documented by part. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware

Requirements · Research details and dates

On this page

Overview

GR-MG (*Leveraging Partially Annotated Data via Multi-Modal Goal-Conditioned Policy*, IEEE RA-L / arXiv:2408.14368) is a two-stage visuomotor manipulation system that decouples language-to-visual-subgoal planning from action execution so that actionless human activity videos (Ego4D) and textless robot trajectories can both contribute to policy learning. The official repository (bytedance/GR-MG at commit f45b0725e462) is licensed under Apache-2.0 and links three publicly downloadable checkpoints verified live via HTTP 200. [1] [2] [3] [4]

Model card summary

Architecture

Two-component system: (1) InstructPix2Pix-based goal image generation model; (2) GR_MG policy combining a ViT-Base MAE image encoder, CLIP text encoder, Perceiver Resampler (depth 3, 4 heads, 9 latents), and 12-layer / 12-head GPT-2 transformer (embed_dim 384, seq_len 10) with a CVAE-style action decoder (act_latent_dim 32) and progress decoder.

Hyperparameters verified directly from policy/config/train.json and policy/model/model.py.

[1]

Base model

InstructPix2Pix (timbrooks/instruct-pix2pix) + MAE ViT-Base + CLIP + GR-1 Ego4D pretraining recipe

[1]

Parameter count

Not stated as a single scalar in README (policy checkpoint is 1.28 GB; Ego4D pretrained checkpoint is 1.24 GB; InstructPix2Pix goal generator checkpoint is 13.75 GB including optimizer/EMA state)

Exact parameter counts are not listed in README.md; checkpoint sizes verified via HTTP HEAD.

[1] [3]

Input modalities
rgbproprioceptionlanguage

Workspace RGB (and optional hand RGB), language instruction, generated goal RGB image, and optional proprioceptive state (use_state flag in GR_MG).

[1]

Output modalities
ee-pose-deltagripper-commandvideo-frames

Goal generator outputs predicted goal RGB image frames; policy predicts 5-step CALVIN action chunks (act_len: 5) plus progress predictions.

[1]

Action space

Continuous 5-step action chunks (act_len: 5, act_latent_dim: 32) over a 10-step observation history (seq_len: 10) for 7-DoF CALVIN manipulation (end-effector delta pose + gripper command).

Verified from policy/config/train.json.

[1]

Trained / supported embodiments
  • Franka Emika Panda (CALVIN simulation benchmark)

[1]

Training datasets
  • Ego4D (human activity videos for policy pretraining)
  • CALVIN (robot manipulation trajectories)

[1] [4]

Interfaces
  • PyTorch Lightning DDP shell runners (goal_gen/train_ip2p.sh, policy/main.sh, evaluate/eval.sh)
  • JSON configuration files (policy/config/train.json, goal_gen/config/train.json)
  • PyTorch Lightning .ckpt and PyTorch .pt checkpoints

[1] [3]

Checkpoints

CheckpointFormatSizeAccessPinned revision
GR-MG Multi-modal Goal Conditioned Policy (epoch=47-step=83712.ckpt)pytorch1.28 GBopenetag:1e47858a7734079f920401ed4391eca0-24
GR-MG Goal Image Generation Model (goal_gen.ckpt)pytorch13.75 GBopenetag:346840e04b852ddccb5b9802cda0090e-262
GR-MG Ego4D Pretrained Policy (pretrained.pt)pytorch1.24 GBopenetag:5a3ccfd28f74ff03c7fd6f47587d1754-23

All three direct download URLs verified HTTP 200 with exact sizes (1.281 GB, 13.754 GB, 1.244 GB) and ETags on 2026-11-21.

[1] [3]

Is this for you?

Use cases
  • Evaluating multi-modal goal-conditioned manipulation on the CALVIN benchmark
  • Fine-tuning a goal-image diffusion model (InstructPix2Pix) for visual subgoal generation
  • Pretraining a GPT-2-based visuomotor policy on Ego4D human activity videos before robot fine-tuning

[1]

Known limitations
  • Tested upstream only with CUDA 12.1 and Python 3.9.
  • Goal image generation checkpoint (goal_gen.ckpt, 13.75 GB) derives from InstructPix2Pix/Stable Diffusion and requires downloading upstream timbrooks/instruct-pix2pix and MAE ViT-Base weights.
  • Reproducing Stage-2 Ego4D pretraining from scratch requires separately obtaining access to the Ego4D dataset; released checkpoints target CALVIN simulation.

[1] [3]

Safety and limitations
  • Checkpoints are hosted as direct object-storage downloads without a separate per-file weight license; downstream use must also respect upstream InstructPix2Pix/Stable Diffusion, MAE, CLIP, and Ego4D terms.
  • Released checkpoints are trained for the simulated CALVIN benchmark and are not directly validated on arbitrary physical robots.
  • Repository is tested only on CUDA 12.1 and Python 3.9 and has had no commits since January 2025.

[1] [2] [3]

Compute & fine-tuning

Inference compute
  • Documented hardware: CUDA 12.1 GPU(s) (exact GPU model not stated in README)
  • Precision: bf16
  • Edge / consumer capable: No

Both train.json configs set precision: bf16; exact minimum VRAM and inference latency_ms are not documented.

[1] · Editorial assessment

Fine-tuning
  • Supported: Yes
  • Methods: Goal image generator training (goal_gen/train_ip2p.sh); Ego4D policy pretraining (policy/main.sh policy/config/pretrain.json); Policy fine-tuning on CALVIN (policy/main.sh policy/config/train.json)
  • Recipe: Upstream fine-tuning documentation

[1]

Host computer

Multi-GPU CUDA 12.1 DDP training configured with BF16 precision (batch_size 64 for goal_gen, 32 for policy); multi-GPU CALVIN evaluation is supported.

[1]

Operating system
  • Linux

CUDA 12.1, bash install scripts, and CALVIN target Linux.

[1] · Editorial assessment

Dependencies
  • Python 3.9
  • CUDA 12.1
  • PyTorch & PyTorch Lightning
  • InstructPix2Pix (timbrooks/instruct-pix2pix)
  • MAE ViT-Base (mae_pretrain_vit_base.pth)
  • CLIP
  • CALVIN simulator and dataset

[1]

Skills
  • Python 3.9
  • PyTorch / PyTorch Lightning DDP
  • CALVIN benchmark setup

Inferred from documented PyTorch Lightning DDP configs and shell scripts.

[1] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1] · Model card / documentation [1]

  • Clone bytedance/GR-MG on a CUDA 12.1 / Python 3.9 Linux environment and run bash ./goal_gen/install.sh and bash ./policy/install.sh.
  • Download pretrained InstructPix2Pix (timbrooks/instruct-pix2pix) into resources/IP2P/ and MAE ViT-Base (mae_pretrain_vit_base.pth) into resources/MAE/.
  • Download and unzip the CALVIN dataset and fetch epoch=47-step=83712.ckpt and goal_gen.ckpt (plus pretrained.pt for training).
  • Run bash ./evaluate/eval.sh ./policy/config/train.json for CALVIN evaluation or bash ./policy/main.sh ./policy/config/train.json for policy training.

[1]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
GR-MG paper evaluations (IEEE RA-L / arXiv:2408.14368; CALVIN and real-robot tasks)See arXiv:2408.14368 (no numeric table in repository README)Upstream-reported paper evaluations; not independently verified or reproduced by OSR.

[1] [4]

Project notes

Two-stage architecture and verified checkpoints

In policy/model/model.py and policy/config/train.json, GR-MG pairs an InstructPix2Pix goal-image generator (goal_gen.ckpt, 13.75 GB) with a multi-modal goal-conditioned policy (epoch=47-step=83712.ckpt, 1.28 GB) initialized from an Ego4D-pretrained checkpoint (pretrained.pt, 1.24 GB). The policy combines a ViT-Base MAE image encoder, CLIP language encoder, Perceiver Resampler (depth: 3, 4 heads, 9 latents), and a 12-layer / 12-head GPT-2 transformer (embed_dim: 384, seq_len: 10), predicting 5-step continuous action chunks (act_len: 5) for CALVIN manipulation through its CVAE-style action decoder (act_latent_dim: 32) while separately predicting task progress through its progress decoder. Exact total parameter counts are not reported in the README. [1] [3]

Setup, compute, and upstream-reported evaluations

The authors note that GR-MG is tested only with CUDA 12.1 and Python 3.9, using PyTorch Lightning DDP with precision: bf16 (batch size 64 for goal_gen, 32 for policy). Setup requires downloading upstream timbrooks/instruct-pix2pix weights, mae_pretrain_vit_base.pth, and the CALVIN dataset, and pretraining from scratch additionally requires obtaining Ego4D access. As an upstream evaluation workflow, the README documents running bash ./evaluate/eval.sh ./policy/config/train.json, with the goal generator and policy checkpoints selected inside eval.sh. The released checkpoints target CALVIN simulation, and the supplied evidence does not establish direct applicability to arbitrary physical robots. Exact minimum GPU VRAM, inference latency, and numeric CALVIN benchmark tables are not included in the repository README (benchmark results are reported in arXiv:2408.14368); Open Source Robotics has not run training, fine-tuning, inference, or CALVIN evaluations for this profile. [1] [4]

What we checked

None. Prepared from public upstream sources without hands-on training, fine-tuning, or simulation execution.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Jan 12, 2025 [1]
Latest release
Aug 26, 2024 [3]
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
bytedance/GR-MG@f45b0725e46291d11ba65c04ae184dc9755bc5ee · software repository

Separate license matrix

Apache-2.0 official implementation of an IEEE RA-L manipulation policy with three publicly downloadable checkpoints (CALVIN policy, InstructPix2Pix goal generator, and Ego4D pretrained policy) verified live.

LayerAvailability / license
Codeavailable · Apache-2.0
Verified Apache License Version 2.0 in root LICENSE file. [1] [2]
Model weightsavailable · No separate weight license file (repo is Apache-2.0; upstream InstructPix2Pix/Stable Diffusion, MAE, CLIP, and Ego4D terms apply)
All three ByteTOS checkpoint downloads (policy, goal_gen, pretrained) are publicly accessible without authentication. [1] [2] [3]
Training datasetsavailable · MIT (CALVIN); Ego4D requires separate Ego4D data license agreement
CALVIN is openly downloadable under MIT; Ego4D requires registration/license. [1]
Documentationavailable · Apache-2.0
[1] [2]

Licensing boundaries

The bytedance/GR-MG codebase is Apache-2.0, and all three ByteTOS checkpoints are directly downloadable without gating. However, the checkpoints do not include a separate per-file weight license, and downstream deployment must account for upstream licenses on InstructPix2Pix (built on Stable Diffusion), MAE ViT-Base, CLIP, CALVIN (MIT), and Ego4D. [1] [2] [3]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from bytedance/GR-MG@f45b0725e462 and HTTP-verified ByteTOS checkpoints.