Open Source RoboticsRobots, software & learning

BAKU

Robotics model

Efficient multi-task transformer visuomotor policy combining a FiLM-conditioned ResNet visual encoder, causal GPT observation trunk, action chunking, and pluggable action heads (deterministic, GMM, BeT, Diffusion, VQ-BeT), with MIT-licensed code and a 2.56 GB OSF benchmark weights archive. [1] [2] [3] [4]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Imitation learning and multi-task policy researchers; LIBERO, Meta-World, and DMControl benchmark practitioners; xArm manipulation labs using Open-Teach teleoperation [1]
Parameters
Not stated as a single scalar in README (compact hidden_dim: 256 multi-task transformer [1] [3]
Embodiments
Franka Emika Panda (LIBERO simulation), Sawyer arm (Meta-World simulation), DMControl continuous-control agents, … [1]
Source rights
Named licenses are documented by part. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware

Requirements · Research details and dates

On this page

Overview

BAKU (siddhanthaldar/BAKU, commit 47e367059191; arXiv:2406.07539) is a compact, sample-efficient multi-task transformer architecture for visuomotor robot policy learning. The MIT-licensed Hydra/PyTorch repository pairs with a publicly accessible 2.56 GB OSF weights archive (weights.zip, node 3x8v5) that upstream describes as weights trained on simulated benchmarks, plus real-robot xarm-env and Open-Teach teleoperation integration for UFactory xArm experiments. [1] [2] [3] [4]

Model card summary

Architecture

FiLM-conditioned ResNet visual encoder (encoder_type: resnet, film: true) + causal GPT transformer observation trunk (policy_type: gpt, hidden_dim: 256) with action chunking (num_queries: 10, temporal_agg: true) and pluggable action heads (deterministic, gmm, bet, diffusion, vqbet).

Verified directly from baku/cfgs/config.yaml and baku/cfgs/agent/baku.yaml.

[1]

Base model

ResNet visual backbone (FiLM-conditioned) + GPT-style policy transformer

[1]

Parameter count

Not stated as a single scalar in README (compact hidden_dim: 256 multi-task transformer; OSF weights.zip bundles multiple benchmark checkpoints totaling 2.56 GB)

Exact per-checkpoint parameter count is not published in README.md or Instructions.md.

[1] [3]

Input modalities
rgbproprioceptionlanguage

Multi-view RGB pixels (or state features for DMControl), optional proprioception (use_proprio: true on LIBERO, false on Meta-World/DMControl), and text/goal/intermediate_goal prompts.

[1]

Output modalities
ee-pose-deltajoint-positionsgripper-command

Continuous action chunks (num_queries: 10 with temporal aggregation) adapted to LIBERO, Meta-World, DMControl, and xArm action spaces.

[1]

Action space

Continuous multi-step action chunks (num_queries: 10, temporal_agg: true) predicted via configurable deterministic, GMM, BeT, Diffusion, or VQ-BeT action heads tailored to each benchmark or xArm suite.

[1]

Trained / supported embodiments
  • Franka Emika Panda (LIBERO simulation)
  • Sawyer arm (Meta-World simulation)
  • DMControl continuous-control agents
  • UFactory xArm (real-world xarm-env + Open-Teach)

[1]

Training datasets
  • LIBERO-90 (public demonstrations converted via generate_libero.py)
  • Meta-World, DMControl, and real-world xArm Kitchen demonstrations (available upon email request to [email protected])

[1]

Interfaces
  • Hydra CLI (train.py, eval.py, eval_ray.py)
  • PyTorch model checkpoints (bc_weight=...)
  • xArm / Open-Teach real-robot environment wrapper (xarm-env)

[1]

Checkpoints

CheckpointFormatSizeAccessPinned revision
BAKU Simulated Benchmarks Weights Archive (weights.zip on OSF)pytorch2.56 GBopen2024-06-11

Verified via OSF API v2 (public node 3x8v5, weights.zip = 2,561,746,368 bytes / 2.562 GB, direct download https://osf.io/download/dvjn2/?view_only=fb8285f025e84d23a41a0eef683a7e6d).

[1] [3]

Is this for you?

Use cases
  • Evaluating pretrained multi-task BAKU policies on LIBERO-90, Meta-World, and DMControl using sequential (eval.py) or Ray-parallel (eval_ray.py) evaluation
  • Training custom multi-task policies with swappable action heads (deterministic, GMM, BeT, Diffusion, VQ-BeT) and action chunking
  • Benchmarking against bundled MT-ACT and RT-1 baselines or deploying on a real xArm with Open-Teach

[1]

Known limitations
  • Upstream describes the OSF weights archive (weights.zip, 2.56 GB) as weights trained on simulated benchmarks rather than real-robot xArm weights (archive contents were not extracted during review).
  • Only LIBERO demonstrations are directly downloadable via upstream scripts; Meta-World, DMControl, and real-world xArm Kitchen demonstration datasets require emailing the author ([email protected]).
  • Supplied materials do not establish a separate license for the OSF weights archive, and the repository's MIT License alone does not resolve weight-specific licensing scope.

[1] [3]

Safety and limitations
  • Upstream describes the OSF weights archive as simulated-benchmark weights (archive contents were not extracted during review), and real-world xArm Kitchen demonstration datasets require emailing the author.
  • Meta-World and DMControl demonstration datasets also require emailing the author rather than direct public download.
  • Supplied materials do not establish a separate license for the OSF weights archive; the repository's MIT License alone does not resolve weight-specific licensing scope.

[1] [2] [3]

Compute & fine-tuning

Inference compute
  • Documented hardware: Single CUDA GPU (exact model not specified)
  • Precision: fp32
  • Edge / consumer capable: Yes

Compact ResNet + hidden_dim 256 GPT policy architecture; exact minimum VRAM and latency_ms are not documented.

[1] · Editorial assessment

Fine-tuning
  • Supported: Yes
  • Methods: Full multi-task training / fine-tuning via Hydra train.py (LIBERO, Meta-World, DMControl, xArm); Checkpoint initialization via load_bc=true bc_weight=...
  • Recipe: Upstream fine-tuning documentation

[1]

Host computer

Single CUDA GPU configured by default (device: cuda, batch_size: 64, hidden_dim: 256); exact GPU model and VRAM requirements are not stated in README or Instructions.md.

[1]

Operating system
  • Linux

CUDA, MuJoCo/LIBERO/Meta-World, and xArm-Python-SDK workflows target Linux.

[1] · Editorial assessment

Dependencies
  • Python (via conda_env.yml)
  • PyTorch
  • Hydra
  • LIBERO / Meta-World / DMControl (for simulation benchmarks)
  • Ray (optional, for eval_ray.py)
  • xArm-Python-SDK & Open-Teach (optional, for real-robot experiments)

[1]

Skills
  • Python
  • PyTorch & Hydra configuration
  • MuJoCo / LIBERO / Meta-World simulation
  • Optional Ray parallel evaluation and xArm-Python-SDK

Inferred from documented conda, Hydra CLI overrides, Ray, and xArm instructions.

[1] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1]

  • Clone siddhanthaldar/BAKU, initialize submodules (Metaworld, Open-Teach), and create the conda environment from conda_env.yml.
  • Install LIBERO (or editable-install Metaworld / xarm-env) and convert LIBERO demonstrations to .pkl via baku/data_generation/generate_libero.py.
  • Set root_dir in cfg/config.yaml and cfg/config_eval.yaml.
  • Train via python train.py agent=baku suite=libero dataloader=libero suite/task=libero_90 suite.hidden_dim=256 or download weights.zip from OSF and run eval.py / eval_ray.py.

[1] [3]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
BAKU paper evaluations (arXiv:2406.07539; LIBERO-90, Meta-World, DMControl, real-world xArm Kitchen)See arXiv:2406.07539 (no numeric table in repository README)Upstream-reported simulation and real-robot evaluations; not independently reproduced by OSR.

[1] [4]

Project notes

Modular encoder, transformer trunk, and swappable action heads

In baku/cfgs/config.yaml and baku/cfgs/agent/baku.yaml, BAKU pairs a FiLM-conditioned ResNet visual encoder (encoder_type: resnet, film: true) with a causal GPT-style observation transformer (policy_type: gpt, hidden_dim: 256) and action chunking (num_queries: 10, temporal_agg: true). The policy supports multi-view RGB pixels (or state features on DMControl), optional proprioceptive state (use_proprio), and text, goal, or intermediate_goal prompts, while letting practitioners swap between deterministic, gmm, bet, diffusion, and vqbet action heads. The codebase also bundles Hydra configs to train MT-ACT and RT-1 baselines under the same evaluation harness, supporting both sequential (eval.py) and Ray-parallel (eval_ray.py) rollouts. [1]

Pretrained weights, compute, and upstream-reported evaluations

Upstream hosts weights.zip (2,561,746,368 bytes) on public OSF node 3x8v5 (OSF node date_modified: 2024-06-11) and describes the archive as weights trained on simulated benchmarks (archive contents were not extracted during review). Training defaults to single-GPU CUDA (batch_size: 64); exact per-checkpoint parameter counts, minimum GPU VRAM, and real-robot control frequencies are not documented in README.md or Instructions.md. Benchmark evaluations across LIBERO-90, Meta-World, DMControl, and a real-world xArm Kitchen setup are reported in arXiv:2406.07539 (no numeric table is printed in the repository README); Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [3] [4]

What we checked

None. Prepared from public upstream sources without hands-on training, fine-tuning, or robot execution.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Mar 16, 2025 [1]
Latest release
Jun 11, 2024 [3]
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
siddhanthaldar/BAKU@47e367059191de30ac4b6353ff5615fdef11d8c5; OSF node 3x8v5 (weights.zip, modified 2024-06-11) · software repository

Separate license matrix

MIT-licensed multi-task visuomotor transformer policy with a publicly downloadable 2.56 GB OSF benchmark weights archive, Hydra training/evaluation recipes, and xArm real-robot integration.

LayerAvailability / license
Codeavailable · MIT
Verified MIT License (Copyright 2025 Siddhant Haldar) in root LICENSE file. [1] [2]
Model weightsavailable · No separate OSF weight license declared (code repository is MIT)
Public OSF archive weights.zip (2.56 GB) for simulated benchmarks verified via OSF API. [1] [2] [3]
Training datasetsavailable · Mixed (LIBERO is publicly downloadable; Meta-World, DMControl, and xArm Kitchen datasets require email request)
[1]
Documentationavailable · MIT
[1] [2]

Licensing and dataset access caveats

The siddhanthaldar/BAKU codebase is MIT-licensed (Copyright 2025 Siddhant Haldar), but the supplied materials do not establish a separate license for the OSF weights archive, and the repository's MIT license alone does not resolve the weights' licensing scope—practitioners needing commercial clarity should confirm weight-specific terms upstream. While LIBERO demonstrations are publicly downloadable via upstream LIBERO tooling and convertible with baku/data_generation/generate_libero.py, access to the authors' Meta-World, DMControl, and real-world xArm Kitchen demonstration datasets requires emailing [email protected]. [1] [2] [3]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from siddhanthaldar/BAKU@47e367059191 and OSF node 3x8v5.