SpatialBot
Robotics model3B-parameter RGB-D spatial Vision-Language Model (ICRA 2025) from BAAI-DCAI and collaborators built on Bunny (Phi-2 + SigLIP) and trained on SpatialQA for metric depth reading and 3D spatial reasoning. [1] [5]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Embodied AI and robotics researchers exploring RGB-D spatial reasoning VLMs and depth-aware visual question answering; Teams fine-tuning compact 3B multimodal models (Phi-2 + SigLIP) on custom RGB-D instruction datasets [1] [5]
- Source rights
- Named licenses are documented by part. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
Overview
SpatialBot (BAAI-DCAI/SpatialBot, ICRA 2025, arXiv:2406.13642) is a 3B-parameter RGB-D Vision-Language Model from BAAI-DCAI and collaborators (SJTU, Stanford, PKU, Oxford, SEU) designed to give compact VLMs metric depth reading and 3D spatial reasoning capabilities. At pinned commit 2d84503eede300e751f4b007052a0b6a5cd274a3 (last updated July 2026), the repository provides Hugging Face Quickstart code, RGB/RGB-D CLI tools, LoRA and full-parameter continuous fine-tuning scripts, and SpatialBench evaluation scripts. [1] [3] [5]
Model card summary
- Architecture
Multi-image extension of the Bunny lightweight VLM architecture pairing a SigLIP vision encoder (supporting CLIP, SigLIP, and EVA-CLIP in training) and cross-modality MLP projector with a Phi-2 base LLM (training also supports Phi-3, Qwen1.5-0.5B/4B, and Llama-3-8B), encoding single-channel depth maps into 3-channel uint8 RGB tensors fed alongside RGB images.
- Base model
Bunny (Microsoft Phi-2 + SigLIP vision tower for released SpatialBot-3B)
Training scripts also support Phi-3, Qwen1.5-0.5B/1.8B(4B), and Llama-3-8B backbones.
- Parameter count
3B parameters (SpatialBot-3B merged safetensors: 6.365 GB across 2 shards; SpatialBot-3B-LoRA adapter: 0.397 GB)
Verified exact Hugging Face shard sizes: SpatialBot-3B (4,990,950,824 + 1,373,673,760 = 6,364,624,584 bytes / 6.365 GB); SpatialBot-3B-LoRA (adapter_model.safetensors 377,539,280 bytes + non_lora_trainables.bin 19,017,904 bytes = 396,557,184 bytes / 0.397 GB).
- Input modalities
- rgbdepthrgbdlanguage
Accepts RGB images, single-channel depth maps encoded into 3-channel uint8 tensors (sensor depth recommended; estimated depth instructions provided), and natural-language prompts.
- Output modalities
- waypoints
Released SpatialBot-3B checkpoints output natural-language spatial answers (including depth values and spatial relations); the README also describes an unreleased embodied VLA variant predicting per-frame delta positions/velocities or keypoints.
- Action space
Released SpatialBot-3B checkpoints are RGB-D vision-language models outputting text responses (depth values and spatial reasoning); the paper/README also describe an embodied VLA variant predicting delta positions/velocities or keypoints, whose checkpoints are not yet released.
- Trained / supported embodiments
- Embodiment-agnostic RGB-D spatial VLM (SpatialBot-3B); SpatialQA-E targets robotic pick-and-place manipulation
- Training datasets
- SpatialQA (multi-level RGB-D spatial reasoning dataset on Hugging Face, gated=auto, CC-BY-4.0)
- SpatialQA-E (embodied manipulation spatial reasoning dataset on Hugging Face, gated=auto, CC-BY-4.0)
- Bunny pre-training/fine-tuning base mixtures (referenced in README)
- Interfaces
- Hugging Face transformers Python API (AutoModelForCausalLM with trust_remote_code=True)
- CLI inference tools (bunny.serve.cli_depth for RGB-D, bunny.serve.cli for RGB)
- Shell scripts for LoRA/full training, LoRA merging, and SpatialBench evaluation
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| RussRobin/SpatialBot-3B (merged Phi-2 + SigLIP RGB-D VLM, auto-gated) | safetensors | 6.365 GB | gated | 41d3b52c6420 |
| RussRobin/SpatialBot-3B-LoRA (LoRA adapter + non_lora_trainables, auto-gated) | safetensors | 0.397 GB | gated | 9b8fabf45c25 |
Both checkpoints verified via Hugging Face API as gated=auto and tagged license:cc-by-4.0 (also mirrored on WiseModel). Embodied SpatialBot VLA checkpoints remain unreleased ('available soon').
Is this for you?
- Use cases
- Point-level depth queries and multi-level spatial reasoning over paired RGB and depth maps (SpatialBot-3B)
- Evaluating VLMs on SpatialBench with either RGB-only or RGB-D inputs
- Continuous LoRA or full-parameter fine-tuning on custom multi-image (0–8 images, 0–2 recommended) instruction datasets
- Embodied pick-and-place research using the SpatialQA-E dataset (note: embodied VLA checkpoints are not yet released)
- Known limitations
- Embodied VLA checkpoints (fine-tuned on SpatialQA-E for pick-and-place delta position/velocity or keypoint prediction) are still marked 'available soon' in the README; only the SpatialBot-3B and SpatialBot-3B-LoRA VLM checkpoints are published.
- Hugging Face checkpoints (RussRobin/SpatialBot-3B, SpatialBot-3B-LoRA) and datasets (SpatialQA, SpatialQA-E, SpatialBench) are auto-gated (gated=auto) and require trust_remote_code=True in transformers.
- License ambiguity in repository: root LICENSE file is MIT, whereas README badges state Apache-2.0 for code and CC-BY-4.0 for weights/data, and README text restricts checkpoints to uses complying with Bunny, LLaMA 3, Phi-2, Phi-3, Qwen-1.5, and GPT-4 terms.
- Open Source Robotics has not installed SpatialBot, accepted Hugging Face gating, downloaded weights, run inference or fine-tuning, or reproduced SpatialBench metrics.
- Safety and limitations
- Released SpatialBot-3B weights are an RGB-D vision-language model outputting natural-language text (not a closed-loop robot controller); embodied VLA weights are unreleased.
- Loading SpatialBot-3B requires trust_remote_code=True in Hugging Face transformers, which executes remote Python modeling code.
- Downstream commercial use requires legal review due to the MIT vs. Apache-2.0 README discrepancy and upstream Bunny/Phi-2/LLaMA-3/Qwen-1.5/GPT-4 compliance restrictions.
- No independent accuracy, latency, or VRAM verification was performed by Open Source Robotics.
Compute & fine-tuning
- Inference compute
- Documented hardware: NVIDIA CUDA GPU (fp16 via device_map='auto'; CPU fp32 fallback shown in Quickstart)
- Precision:
fp16 (fp32 on CPU) - Edge / consumer capable: No
Exact minimum VRAM and generation latency are not stated in the README (5.927 GB safetensors on disk).
- Fine-tuning
- Supported: Yes
- Methods: LoRA fine-tuning (script/train/finetune_lora.sh); full-parameter continuous fine-tuning (script/train/finetune_full.sh)
- Recipe: Upstream fine-tuning documentation
Supports pretraining, LoRA fine-tuning, LoRA merging, and continuous fine-tuning on custom 0–8 image JSON conversations.
- Host computer
NVIDIA CUDA GPU recommended for fp16 inference (6.365 GB safetensors on disk; CPU fp32 fallback supported); Docker image provided.
- Operating system
- Linux
Docker and shell training/evaluation scripts target Linux.
[1] · Editorial assessment
- Dependencies
- PyTorch, Hugging Face transformers, accelerate, Pillow, NumPy
- Bunny package (replaced by SpatialBot pip install -e .)
- Optional: Docker (russellrobin/bunny:latest)
- Skills
- Python, PyTorch, and Hugging Face transformers/accelerate
- RGB-D depth encoding and LoRA/Bunny VLM fine-tuning
Inferred from Quickstart, CLI depth inference, and continuous fine-tuning instructions.
[1] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [1]
- Install torch, transformers, accelerate, pillow, and numpy (or pull the russellrobin/bunny:latest Docker image and install SpatialBot via pip install -e .).
- Accept the auto-gated Hugging Face terms for RussRobin/SpatialBot-3B (or SpatialBot-3B-LoRA).
- Load via AutoModelForCausalLM.from_pretrained('RussRobin/SpatialBot-3B', torch_dtype=torch.float16, device_map='auto', trust_remote_code=True), pack single-channel depth into 3-channel uint8 RGB, and pass [rgb, depth] to model.generate.
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| SpatialBench, general VLM benchmarks, and Embodied AI tasks (ICRA 2025 / arXiv:2406.13642) | Upstream abstract reports substantial improvements on SpatialBench, general VLM benchmarks, and Embodied AI tasks when trained on SpatialQA (detailed numeric tables are in the ICRA 2025 / arXiv paper rather than the README). | Upstream-reported in ICRA 2025 / arXiv:2406.13642 abstract; OSR has not run SpatialBench or reproduced paper numbers. |
Project notes
Architecture, checkpoints, and VLM vs. VLA scope
SpatialBot extends the lightweight Bunny multimodal framework to multi-image RGB-D inputs (supporting 0–8 images, with 0–2 recommended). The released SpatialBot-3B model pairs a SigLIP vision tower and cross-modality MLP projector with a Phi-2 base LLM (training scripts also support Phi-3, Qwen1.5-0.5B/4B, Llama-3-8B, CLIP, and EVA-CLIP). To feed depth through a standard vision encoder without architectural changes, single-channel depth images are bit-packed into a 3-channel uint8 RGB tensor and passed alongside the RGB frame. [1]
Two auto-gated (gated=auto) checkpoints are verified on Hugging Face: RussRobin/SpatialBot-3B (merged model, 6.365 GB across two safetensors shards, revision 41d3b52c642058dfb087885bec0b8e37e0e67f8d, loaded in the documented Quickstart via transformers with trust_remote_code=True) and RussRobin/SpatialBot-3B-LoRA (0.397 GB total for adapter_model.safetensors plus non_lora_trainables.bin, revision 9b8fabf45c2579477669daa5a93e90fdfcd6e5d4, which requires base Phi-2/SigLIP weights and editing local paths in config.json per the README). Minimum GPU VRAM and inference latency are unspecified and unmeasured (file sizes on disk do not establish runtime VRAM). Importantly, while the README also describes an Embodied SpatialBot VLA variant fine-tuned on SpatialQA-E to predict per-frame delta positions/velocities or keypoints for pick-and-place, its checkpoints remain marked "available soon" and are not yet released; only the RGB-D spatial VLM weights are currently published. [1] [4]
Upstream-reported benchmarks
The ICRA 2025 / arXiv:2406.13642 abstract reports substantial improvements on SpatialBench, general VLM benchmarks, and Embodied AI tasks when training on SpatialQA (detailed numeric tables appear in the paper rather than the README). All claims are upstream-reported; Open Source Robotics has not installed SpatialBot, accepted Hugging Face gating, downloaded weights, run training, fine-tuning, or inference, or reproduced SpatialBench evaluations. [1] [5]
What we checked
Source researched only; no installation, training, fine-tuning, inference, benchmark, or physical hardware test performed. No affiliate links or reused media.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Upstream last updated
- Jul 26, 2026 [3]
- Latest release
- Not established
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- commit snapshot; no GitHub release / 2d84503eede300e751f4b007052a0b6a5cd274a3 · software repository
Separate license matrix
Include SpatialBot as a 3B RGB-D spatial VLM with public training/evaluation code and datasets, marked partial because HF checkpoints/datasets are auto-gated with upstream LLM/GPT-4 license caveats, root LICENSE (MIT) conflicts with the README badge (Apache-2.0), and embodied VLA weights are unreleased.
| Layer | Availability / license |
|---|---|
| Code | available · MIT (root LICENSE file; README badge inconsistently labels Code License as Apache-2.0) Root LICENSE file contains MIT License (Copyright 2024 BAAI-DCAI), while README badge says Apache_2.0 and links ./LICENSE. [1] [2] [3] |
| Model weights | available · CC-BY-4.0 (auto-gated on HF) plus upstream Bunny/Phi-2/Phi-3/LLaMA-3/Qwen-1.5/GPT-4 terms SpatialBot-3B and SpatialBot-3B-LoRA are auto-gated on Hugging Face under CC-BY-4.0 with additional upstream LLM/GPT-4 compliance conditions; embodied VLA checkpoints are not yet released. [1] [4] |
| Training datasets | available · CC-BY-4.0 (SpatialQA, SpatialQA-E, SpatialBench; auto-gated on HF) SpatialQA, SpatialQA-E, and SpatialBench are all verified on Hugging Face as gated=auto and tagged license:cc-by-4.0. [1] [6] |
| Documentation | available · MIT (root LICENSE) README, depthmap preparation guide, LoRA/full evaluation guides, and merge instructions included. [1] [2] |
Licensing discrepancies and gating
Careful license review is warranted before commercial use. The repository's root LICENSE file is MIT (Copyright 2024 BAAI-DCAI), yet the README badge labels the code license Apache-2.0 while linking to that same ./LICENSE file. The checkpoints and all three Hugging Face datasets (SpatialQA, SpatialQA-E, SpatialBench) are auto-gated and tagged CC-BY-4.0, and the README further states that checkpoint use must comply with the license agreements of Bunny, LLaMA 3, Phi-2, Phi-3, Qwen-1.5, and GPT-4. [1] [2] [4] [6]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 SpatialBot README
ICRA 2025 overview, RGB-D 3-channel depth encoding Quickstart, Bunny/Phi-2+SigLIP architecture, continuous fine-tuning guide, unreleased embodied VLA note, and license badges/caveats.Revision, date & where cited
github.com · revision
2d84503· Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 5 more - S002 SpatialBot root LICENSE (MIT)
MIT License (Copyright 2024 BAAI-DCAI), contrasting with README's Apache-2.0 code badge.Revision, date & where cited
github.com · revision
2d84503· Accessed Nov 21, 2026
Used for: Is this for you?, Separate license matrix - S003 GitHub repository metadata and pinned commit
Last commit 2026-07-26; no GitHub releases; MIT detected by GitHub; not archived.Revision, date & where cited
BAAI-DCAI/SpatialBot / GitHub · github.com · revision
2d84503· main / · Accessed Nov 21, 2026
Used for: Overview, What we checked, Separate license matrix - S004 Hugging Face SpatialBot-3B and SpatialBot-3B-LoRA metadata
Verified RussRobin/SpatialBot-3B (gated=auto, cc-by-4.0, 5.927 GB across 2 safetensors shards, sha 41d3b52c642058dfb087885bec0b8e37e0e67f8d) and SpatialBot-3B-LoRA (gated=auto, cc-by-4.0, 0.370 GB, sha 9b8fabf45c2579477669daa5a93e90fdfcd6e5d4).Revision, date & where cited
RussRobin / BAAI-DCAI / Hugging Face · huggingface.co · API metadata 2026-11-21 · Accessed Nov 21, 2026
Used for: Model card summary, Checkpoints, Is this for you?, Compute & fine-tuning and 3 more - S005 SpatialBot: Precise Spatial Understanding with Vision Language Models (ICRA 2025 / arXiv:2406.13642)
Introduces RGB-D SpatialBot VLM, SpatialQA dataset, and SpatialBench evaluation for 3D spatial understanding and embodied AI.Revision, date & where cited
SJTU, Stanford, BAAI, PKU, Oxford, SEU (ICRA 2025 / arXiv) · arxiv.org · arXiv:2406.13642v7 abstract · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Upstream-reported benchmarks - S006 Hugging Face SpatialQA, SpatialQA-E, and SpatialBench dataset metadata
Verified SpatialQA, SpatialQA-E, and SpatialBench datasets on Hugging Face (all gated=auto, license:cc-by-4.0).Revision, date & where cited
RussRobin / BAAI-DCAI / Hugging Face · huggingface.co · API metadata 2026-11-21 · Accessed Nov 21, 2026
Used for: Model card summary, Is this for you?, Separate license matrix
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile for SpatialBot (2 verified auto-gated HF checkpoints, SpatialQA/SpatialQA-E/SpatialBench metadata, license discrepancy audit).