BAKU
Robotics modelEfficient multi-task transformer visuomotor policy combining a FiLM-conditioned ResNet visual encoder, causal GPT observation trunk, action chunking, and pluggable action heads (deterministic, GMM, BeT, Diffusion, VQ-BeT), with MIT-licensed code and a 2.56 GB OSF benchmark weights archive. [1] [2] [3] [4]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Imitation learning and multi-task policy researchers; LIBERO, Meta-World, and DMControl benchmark practitioners; xArm manipulation labs using Open-Teach teleoperation [1]
- Parameters
- Not stated as a single scalar in README (compact hidden_dim: 256 multi-task transformer [1] [3]
- Embodiments
- Franka Emika Panda (LIBERO simulation), Sawyer arm (Meta-World simulation), DMControl continuous-control agents, … [1]
- Source rights
- Named licenses are documented by part. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Overview
BAKU (siddhanthaldar/BAKU, commit 47e367059191; arXiv:2406.07539) is a compact, sample-efficient multi-task transformer architecture for visuomotor robot policy learning. The MIT-licensed Hydra/PyTorch repository pairs with a publicly accessible 2.56 GB OSF weights archive (weights.zip, node 3x8v5) that upstream describes as weights trained on simulated benchmarks, plus real-robot xarm-env and Open-Teach teleoperation integration for UFactory xArm experiments. [1] [2] [3] [4]
Model card summary
- Architecture
FiLM-conditioned ResNet visual encoder (encoder_type: resnet, film: true) + causal GPT transformer observation trunk (policy_type: gpt, hidden_dim: 256) with action chunking (num_queries: 10, temporal_agg: true) and pluggable action heads (deterministic, gmm, bet, diffusion, vqbet).
Verified directly from baku/cfgs/config.yaml and baku/cfgs/agent/baku.yaml.
- Base model
ResNet visual backbone (FiLM-conditioned) + GPT-style policy transformer
- Parameter count
Not stated as a single scalar in README (compact hidden_dim: 256 multi-task transformer; OSF weights.zip bundles multiple benchmark checkpoints totaling 2.56 GB)
Exact per-checkpoint parameter count is not published in README.md or Instructions.md.
- Input modalities
- rgbproprioceptionlanguage
Multi-view RGB pixels (or state features for DMControl), optional proprioception (use_proprio: true on LIBERO, false on Meta-World/DMControl), and text/goal/intermediate_goal prompts.
- Output modalities
- ee-pose-deltajoint-positionsgripper-command
Continuous action chunks (num_queries: 10 with temporal aggregation) adapted to LIBERO, Meta-World, DMControl, and xArm action spaces.
- Action space
Continuous multi-step action chunks (num_queries: 10, temporal_agg: true) predicted via configurable deterministic, GMM, BeT, Diffusion, or VQ-BeT action heads tailored to each benchmark or xArm suite.
- Trained / supported embodiments
- Franka Emika Panda (LIBERO simulation)
- Sawyer arm (Meta-World simulation)
- DMControl continuous-control agents
- UFactory xArm (real-world xarm-env + Open-Teach)
- Training datasets
- LIBERO-90 (public demonstrations converted via generate_libero.py)
- Meta-World, DMControl, and real-world xArm Kitchen demonstrations (available upon email request to [email protected])
- Interfaces
- Hydra CLI (train.py, eval.py, eval_ray.py)
- PyTorch model checkpoints (bc_weight=...)
- xArm / Open-Teach real-robot environment wrapper (xarm-env)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| BAKU Simulated Benchmarks Weights Archive (weights.zip on OSF) | pytorch | 2.56 GB | open | 2024-06-11 |
Verified via OSF API v2 (public node 3x8v5, weights.zip = 2,561,746,368 bytes / 2.562 GB, direct download https://osf.io/download/dvjn2/?view_only=fb8285f025e84d23a41a0eef683a7e6d).
Is this for you?
- Use cases
- Evaluating pretrained multi-task BAKU policies on LIBERO-90, Meta-World, and DMControl using sequential (eval.py) or Ray-parallel (eval_ray.py) evaluation
- Training custom multi-task policies with swappable action heads (deterministic, GMM, BeT, Diffusion, VQ-BeT) and action chunking
- Benchmarking against bundled MT-ACT and RT-1 baselines or deploying on a real xArm with Open-Teach
- Known limitations
- Upstream describes the OSF weights archive (weights.zip, 2.56 GB) as weights trained on simulated benchmarks rather than real-robot xArm weights (archive contents were not extracted during review).
- Only LIBERO demonstrations are directly downloadable via upstream scripts; Meta-World, DMControl, and real-world xArm Kitchen demonstration datasets require emailing the author ([email protected]).
- Supplied materials do not establish a separate license for the OSF weights archive, and the repository's MIT License alone does not resolve weight-specific licensing scope.
- Safety and limitations
- Upstream describes the OSF weights archive as simulated-benchmark weights (archive contents were not extracted during review), and real-world xArm Kitchen demonstration datasets require emailing the author.
- Meta-World and DMControl demonstration datasets also require emailing the author rather than direct public download.
- Supplied materials do not establish a separate license for the OSF weights archive; the repository's MIT License alone does not resolve weight-specific licensing scope.
Compute & fine-tuning
- Inference compute
- Documented hardware: Single CUDA GPU (exact model not specified)
- Precision:
fp32 - Edge / consumer capable: Yes
Compact ResNet + hidden_dim 256 GPT policy architecture; exact minimum VRAM and latency_ms are not documented.
[1] · Editorial assessment
- Fine-tuning
- Supported: Yes
- Methods: Full multi-task training / fine-tuning via Hydra train.py (LIBERO, Meta-World, DMControl, xArm); Checkpoint initialization via load_bc=true bc_weight=...
- Recipe: Upstream fine-tuning documentation
- Host computer
Single CUDA GPU configured by default (device: cuda, batch_size: 64, hidden_dim: 256); exact GPU model and VRAM requirements are not stated in README or Instructions.md.
- Operating system
- Linux
CUDA, MuJoCo/LIBERO/Meta-World, and xArm-Python-SDK workflows target Linux.
[1] · Editorial assessment
- Dependencies
- Python (via conda_env.yml)
- PyTorch
- Hydra
- LIBERO / Meta-World / DMControl (for simulation benchmarks)
- Ray (optional, for eval_ray.py)
- xArm-Python-SDK & Open-Teach (optional, for real-robot experiments)
- Skills
- Python
- PyTorch & Hydra configuration
- MuJoCo / LIBERO / Meta-World simulation
- Optional Ray parallel evaluation and xArm-Python-SDK
Inferred from documented conda, Hydra CLI overrides, Ray, and xArm instructions.
[1] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [1]
- Clone siddhanthaldar/BAKU, initialize submodules (Metaworld, Open-Teach), and create the conda environment from conda_env.yml.
- Install LIBERO (or editable-install Metaworld / xarm-env) and convert LIBERO demonstrations to .pkl via baku/data_generation/generate_libero.py.
- Set root_dir in cfg/config.yaml and cfg/config_eval.yaml.
- Train via python train.py agent=baku suite=libero dataloader=libero suite/task=libero_90 suite.hidden_dim=256 or download weights.zip from OSF and run eval.py / eval_ray.py.
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| BAKU paper evaluations (arXiv:2406.07539; LIBERO-90, Meta-World, DMControl, real-world xArm Kitchen) | See arXiv:2406.07539 (no numeric table in repository README) | Upstream-reported simulation and real-robot evaluations; not independently reproduced by OSR. |
Project notes
Modular encoder, transformer trunk, and swappable action heads
In baku/cfgs/config.yaml and baku/cfgs/agent/baku.yaml, BAKU pairs a FiLM-conditioned ResNet visual encoder (encoder_type: resnet, film: true) with a causal GPT-style observation transformer (policy_type: gpt, hidden_dim: 256) and action chunking (num_queries: 10, temporal_agg: true). The policy supports multi-view RGB pixels (or state features on DMControl), optional proprioceptive state (use_proprio), and text, goal, or intermediate_goal prompts, while letting practitioners swap between deterministic, gmm, bet, diffusion, and vqbet action heads. The codebase also bundles Hydra configs to train MT-ACT and RT-1 baselines under the same evaluation harness, supporting both sequential (eval.py) and Ray-parallel (eval_ray.py) rollouts. [1]
Pretrained weights, compute, and upstream-reported evaluations
Upstream hosts weights.zip (2,561,746,368 bytes) on public OSF node 3x8v5 (OSF node date_modified: 2024-06-11) and describes the archive as weights trained on simulated benchmarks (archive contents were not extracted during review). Training defaults to single-GPU CUDA (batch_size: 64); exact per-checkpoint parameter counts, minimum GPU VRAM, and real-robot control frequencies are not documented in README.md or Instructions.md. Benchmark evaluations across LIBERO-90, Meta-World, DMControl, and a real-world xArm Kitchen setup are reported in arXiv:2406.07539 (no numeric table is printed in the repository README); Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [3] [4]
What we checked
None. Prepared from public upstream sources without hands-on training, fine-tuning, or robot execution.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
Separate license matrix
MIT-licensed multi-task visuomotor transformer policy with a publicly downloadable 2.56 GB OSF benchmark weights archive, Hydra training/evaluation recipes, and xArm real-robot integration.
| Layer | Availability / license |
|---|---|
| Code | available · MIT Verified MIT License (Copyright 2025 Siddhant Haldar) in root LICENSE file. [1] [2] |
| Model weights | available · No separate OSF weight license declared (code repository is MIT) Public OSF archive weights.zip (2.56 GB) for simulated benchmarks verified via OSF API. [1] [2] [3] |
| Training datasets | available · Mixed (LIBERO is publicly downloadable; Meta-World, DMControl, and xArm Kitchen datasets require email request) [1] |
| Documentation | available · MIT [1] [2] |
Licensing and dataset access caveats
The siddhanthaldar/BAKU codebase is MIT-licensed (Copyright 2025 Siddhant Haldar), but the supplied materials do not establish a separate license for the OSF weights archive, and the repository's MIT license alone does not resolve the weights' licensing scope—practitioners needing commercial clarity should confirm weight-specific terms upstream. While LIBERO demonstrations are publicly downloadable via upstream LIBERO tooling and convertible with baku/data_generation/generate_libero.py, access to the authors' Meta-World, DMControl, and real-world xArm Kitchen demonstration datasets requires emailing [email protected]. [1] [2] [3]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 siddhanthaldar/BAKU GitHub repository README.md, Instructions.md, and baku/cfgs/{config.yaml,agent/baku.yaml}
Documents FiLM-ResNet + GPT transformer (hidden_dim 256, num_queries 10, temporal_agg true, pluggable deterministic/gmm/bet/diffusion/vqbet heads), LIBERO/Meta-World/DMControl/xArm setup, OSF weights link, and email-gated dataset access for non-LIBERO demos.Revision, date & where cited
siddhanthaldar (GitHub) · github.com · revision
47e3670· Commit 47e367059191 (2025-03-16) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 7 more - S002 siddhanthaldar/BAKU root LICENSE file
MIT License, Copyright (c) 2025 Siddhant Haldar.Revision, date & where cited
siddhanthaldar (GitHub) · github.com · revision
47e3670· LICENSE at commit 47e367059191 · Accessed Nov 21, 2026
Used for: Overview, Is this for you?, Separate license matrix - S003 OSF API v2 verification: BAKU simulated benchmark weights (node 3x8v5)
Public node containing weights.zip (2,561,746,368 bytes / 2.562 GB).Revision, date & where cited
Open Science Framework (OSF) · osf.io · OSF API v2 nodes/3x8v5/files/osfstorage (date_modified 2024-06-11) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 4 more - S004 arXiv:2406.07539 — BAKU: An Efficient Transformer for Multi-Task Policy Learning
Introduces BAKU as a sample-efficient multi-task transformer architecture for visuomotor robot policy learning.Revision, date & where cited
arXiv · arxiv.org · arXiv:2406.07539v2 (2024-06-11) · Accessed Nov 21, 2026
Used for: Overview, Upstream-reported benchmarks, Pretrained weights, compute, and upstream-reported evaluations
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from siddhanthaldar/BAKU@47e367059191 and OSF node 3x8v5.