EnerVerse-AC
Robotics modelAction-conditional latent video diffusion world model (EVAC) from AgiBot for generating 320x512 robotic manipulation rollouts conditioned on robot actions and camera ray maps; strictly noncommercial CC BY-NC-SA 4.0 single-view inference repo and AgiBot-World-only checkpoint. [1] [2] [3] [4]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Robotic manipulation and world-model researchers; AgiBot-World dataset and IROS 2025 challenge participants; Noncommercial embodied policy evaluation researchers [2] [3] [5]
- Parameters
- Not explicitly stated upstream (6.38 GB DeepSpeed PyTorch checkpoint EnerV_AC_deepspeed_v0.1.pt [2] [3]
- Embodiments
- AgiBot-World bimanual manipulation robot (single-view head camera in released checkpoint) [2] [4]
- Source rights
- Some licenses are unresolved. Includes noncommercial terms. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- License
- CC BY-NC-SA 4.0 (stated in README; no root LICENSE file) (noncommercial), CC BY-NC-SA 4.0 (noncommercial) (noncommercial), CC BY-NC-SA 4.0 (noncommercial) · license not fully assessed for datasets · Details by part
Overview
EnerVerse-AC (EVAC), developed by AgiBot (AgibotTech), is an action-conditional latent video diffusion world model (ACWMLatentDiffusion) built on the LVDM / DynamiCrafter 3D causal UNet architecture with an AutoencoderKL, a frozen OpenCLIP ViT-H/14 image embedder, Resampler action and image projectors, sparse memory, and 6-channel camera ray map encoding. Conditioned on an initial RGB frame, delta robot actions, and camera extrinsics/intrinsics, the released configuration generates 16-frame chunks at 320x512 resolution (15 FPS default save rate) from a single head-camera view. [1] [2] [4] [5]
Model card summary
- Architecture
Action-conditional latent video diffusion world model (ACWMLatentDiffusion, built on LVDM/DynamiCrafter 3D causal UNet with AutoencoderKL, frozen OpenCLIP ViT-H/14 image embedder, Resampler action/image projectors, sparse memory, and 6-channel camera ray map encoding) generating 16-frame chunks at 320x512 resolution.
- Base model
LVDM / DynamiCrafter 3D causal UNet architecture + frozen laion/CLIP-ViT-H-14-laion2B-s32B-b79K
- Parameter count
Not explicitly stated upstream (6.38 GB DeepSpeed PyTorch checkpoint EnerV_AC_deepspeed_v0.1.pt; requires separate laion/CLIP-ViT-H-14-laion2B-s32B-b79K weights)
- Input modalities
- rgbproprioception
Initial RGB frame, delta action trajectories (action_dim=7 per arm, embedding_dim=14 in Resampler), and camera extrinsics/intrinsics encoded as 6-channel ray maps.
- Output modalities
- video-frames
Single-view 320x512 RGB video rollouts (16-frame chunks, save_fps=15 in config).
- Action space
Delta robot action trajectories (cond_stage_key: delta_action; action_dim=7, Resampler embedding_dim=14) paired with camera extrinsics and intrinsics conditioning future video frame generation rather than directly outputting motor commands.
- Control frequency
15 Hz default save rate (save_fps: 15 in configs/agibotworld/config.yaml; offline chunked video generation)
- Trained / supported embodiments
- AgiBot-World bimanual manipulation robot (single-view head camera in released checkpoint)
- Training datasets
- AgiBot-World open-source dataset only (released checkpoint excludes commercially restricted datasets and failure trajectories referenced in the paper)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| agibot-world/EnerVerse-AC (single-view, EnerV_AC_deepspeed_v0.1.pt, CC BY-NC-SA 4.0) | pytorch | 6.38 GB | open | ca05e55b9602 |
Verified via HF API: EnerV_AC_deepspeed_v0.1.pt is 6,384,255,240 bytes (~6.38 decimal GB); ungated but strictly noncommercial CC BY-NC-SA 4.0. Requires separate laion/CLIP-ViT-H-14-laion2B-s32B-b79K download.
Is this for you?
- Use cases
- Single-view action-conditioned robotic manipulation video rollout generation
- Offline visual evaluation of manipulation policies without physical robots
- Baseline world model for the AgiBot World Challenge @ IROS 2025
- Known limitations
- Strictly noncommercial CC BY-NC-SA 4.0 license stated in README and HF model card, and the GitHub repository lacks a root LICENSE file
- Released checkpoint is single-view only (head camera) and trained exclusively on open-source AgiBot-World without the failure-trajectory data or restricted commercial datasets described in the paper
- Main repository provides minimal inference code only (training code is linked separately in AgibotTech/AgiBotWorldChallengeIROS2025-WorldModelBaseline)
- Requires manually downloading laion/CLIP-ViT-H-14-laion2B-s32B-b79K and installing a CUDA 12.1 PyTorch3D wheel alongside PyTorch 2.4.0
- Safety and limitations
- Generates action-conditioned RGB video rollouts for offline evaluation rather than physical robot control commands; the released checkpoint excludes failure trajectories, which may bias policy evaluation toward successful outcomes.
- Strictly noncommercial CC BY-NC-SA 4.0 license (stated in README and HF model card, with no root LICENSE file in GitHub).
- Released checkpoint is single-view head-camera only, differing from the multi-view setup described in the paper.
- Training code is omitted from the main repository, and minimum GPU VRAM / inference latency are unspecified.
Compute & fine-tuning
- Inference compute
- Documented hardware: Not specified upstream (tested with CUDA 11.8+, PyTorch 2.4.0, fp16 Lightning precision)
- Precision:
fp16 - Edge / consumer capable: No
Exact minimum GPU VRAM and generation latency are not stated in the README or config.
- Fine-tuning
- Supported: No
AgibotTech/EnerVerse-AC contains minimal inference code only; README directs users seeking minimal training code to the separate AgibotTech/AgiBotWorldChallengeIROS2025-WorldModelBaseline repository.
- Host computer
NVIDIA CUDA GPU host (tested with CUDA 11.8+, PyTorch 2.4.0, fp16 Lightning precision in config); exact minimum GPU VRAM is not stated upstream.
- Operating system
- Linux
Inferred from CUDA/PyTorch3D Linux wheel and Conda setup instructions.
[2] · Editorial assessment
- Dependencies
- Python 3.10.4
- PyTorch 2.4.0 and CUDA 11.8+ (PyTorch3D wheel built on CUDA 12.1)
- PyTorch Lightning / LVDM codebase dependencies
- laion/CLIP-ViT-H-14-laion2B-s32B-b79K (separate external download)
Setup & usage
Open the upstream getting-started guide [2] · Model card / documentation [2] [5]
- Clone AgibotTech/EnerVerse-AC, create a Python 3.10.4 Conda environment, install requirements.txt (PyTorch 2.4.0), and install the prebuilt PyTorch3D wheel (py310_cu121_pyt240)
- Download EnerV_AC_deepspeed_v0.1.pt from agibot-world/EnerVerse-AC and open_clip_pytorch_model.bin from laion/CLIP-ViT-H-14-laion2B-s32B-b79K, updating paths in configs/agibotworld/config.yaml
- Prepare start frame, action, extrinsic, and intrinsic files (or use tools/prepare_infer_data.py / examples/examples0)
- Run python main/generate_video_acwm.py with --ckp_path and --config_path
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| EnerVerse-AC robotic manipulation evaluation (arXiv:2505.09723v1) & HF validation metrics | Upstream paper abstract reports high-fidelity action-conditioned video generation for policy testing and data augmentation; HF repo also includes EVAC_validation_set_metrics.csv (not summarized in README). | Upstream-reported paper claims apply to the full paper model (which included restricted data and failure trajectories not present in the released AgiBot-World-only single-view checkpoint); not independently verified by OSR. |
Project notes
Scope gap between the paper and the released checkpoint
Adopters should note a material gap between the paper (arXiv:2505.09723v1) and the released weights. While the paper presents multi-view generation and highlights expanding training data with diverse failure trajectories for realistic offline policy evaluation, both the GitHub README and Hugging Face model card state that—due to commercial restrictions on portions of the paper's training data—the released checkpoint (EnerV_AC_deepspeed_v0.1.pt, 6.38 GB) is single-view only, was trained exclusively on the open-source AgiBot-World dataset, and excludes failure trajectory data. As an editorial implication, adopters should not assume that the released checkpoint reproduces the paper's multi-view capabilities or failure-sensitive evaluation performance, and the supplied documentation does not quantify the effect of excluding failure trajectories. Moreover, AgibotTech/EnerVerse-AC provides minimal inference code only; minimal training code is hosted separately in AgibotTech/AgiBotWorldChallengeIROS2025-WorldModelBaseline. [2] [3] [5]
Setup constraints and upstream-reported evaluation
The upstream-tested environment uses Python 3.10.4, CUDA 11.8 (higher CUDA versions noted as expected to work), and PyTorch 2.4.0, while the documented installation recipe installs a prebuilt PyTorch3D wheel built on CUDA 12.1 alongside a separate manual download of laion/CLIP-ViT-H-14-laion2B-s32B-b79K wired into configs/agibotworld/config.yaml. Exact parameter counts, minimum GPU VRAM, and generation latency are not specified in the repository README or config. Although the paper reports high-fidelity action-conditioned video generation for policy testing and the Hugging Face repository includes EVAC_validation_set_metrics.csv, those results are upstream-reported (and paper claims reflect the broader non-public training set); Open Source Robotics has not run training, fine-tuning, or inference on EnerVerse-AC. [2] [3] [4] [5]
What we checked
None. Prepared from public upstream sources without hands-on training, fine-tuning, or inference.
Updates
- Upstream last updated
- Jul 14, 2025 [1]
- Latest release
- Jun 13, 2025 [3]
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- AgibotTech/EnerVerse-AC@fffbdc364009ba30080c837b79f125e8c7d2d2d1; HF agibot-world/EnerVerse-AC@ca05e55b9602666796d6591a981c7ec50d27fc58 · software repository
Separate license matrix
Notable embodied action-conditioned video world model trained on AgiBot-World and used as the IROS 2025 challenge baseline; marked include_partial because both code and weights are strictly noncommercial CC BY-NC-SA 4.0 (with no root LICENSE file), only single-view inference without failure trajectories is released, and training code lives in a separate repo.
| Layer | Availability / license |
|---|---|
| Code | available · CC BY-NC-SA 4.0 (stated in README; no root LICENSE file) Minimal single-view inference code only; GitHub API confirms no root LICENSE file exists, while README declares CC BY-NC-SA 4.0. [1] [2] |
| Model weights | available · CC BY-NC-SA 4.0 (noncommercial) Ungated 6.38 GB single-view checkpoint under noncommercial CC BY-NC-SA 4.0; trained only on open-source AgiBot-World without failure trajectories. [3] |
| Training datasets | available · License not fully assessed Released checkpoint was trained exclusively on open-source AgiBot-World (external upstream license not separately audited here); commercially restricted datasets and failure trajectories from the paper are not released. [2] [3] |
| Documentation | available · CC BY-NC-SA 4.0 [2] |
Noncommercial licensing and repository hygiene
Both the GitHub README and the Hugging Face repository declare CC BY-NC-SA 4.0 (strictly noncommercial, share-alike) for all code and weights, even though the GitHub repository (AgibotTech/EnerVerse-AC, HEAD fffbdc364009, 2025-07-14, with no commits since July 2025) does not contain a root LICENSE file. Readers considering commercial use should seek separate permission rather than treating the missing root LICENSE as permissive, and teams requiring a formal license record should request clarification from the maintainer. [1] [2] [3]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 AgibotTech/EnerVerse-AC GitHub metadata and HEAD commit
Default branch main HEAD is fffbdc364009ba30080c837b79f125e8c7d2d2d1 (2025-07-14T03:53:07Z); archived=false; no root LICENSE file detected by GitHub API.Revision, date & where cited
AgibotTech (GitHub) · github.com · revision
fffbdc3· GitHub API repos/AgibotTech/EnerVerse-AC at fffbdc364009 · Accessed Nov 21, 2026
Used for: Overview, Is this for you?, What we checked, Separate license matrix - S002 AgibotTech/EnerVerse-AC README.md at pinned commit
Documents minimal single-view inference setup (Python 3.10.4, PyTorch 2.4.0, PyTorch3D cu121 wheel, external CLIP ViT-H-14), AgiBot-World-only training caveat (no failure trajectories), link to separate IROS 2025 challenge training repo, and CC BY-NC-SA 4.0 license.Revision, date & where cited
AgibotTech (GitHub) · github.com · revision
fffbdc3· README.md at commit fffbdc364009 · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 6 more - S003 Hugging Face API metadata and model card: agibot-world/EnerVerse-AC
Contains EnerV_AC_deepspeed_v0.1.pt (6,384,255,240 bytes, ~6.38 GB), EVAC_validation_set_metrics.csv, gated=false, license='cc-by-nc-sa-4.0', lastModified 2025-06-13.Revision, date & where cited
agibot-world (Hugging Face) · huggingface.co · HF API /api/models/agibot-world/EnerVerse-AC at revision ca05e55b9602 · Accessed Nov 21, 2026
Used for: Model card summary, Checkpoints, Is this for you?, Setup & usage and 5 more - S004 EnerVerse-AC model configuration (configs/agibotworld/config.yaml)
Defines ACWMLatentDiffusion (LVDM 3D causal UNet, AutoencoderKL, frozen OpenCLIP image embedder, Resampler with embedding_dim=14, 6-channel raymap, sparse memory, n_view=1 head camera, 320x512 resolution, chunk=16, save_fps=15, fp16 Lightning precision).Revision, date & where cited
AgibotTech (GitHub) · github.com · revision
fffbdc3· configs/agibotworld/config.yaml at commit fffbdc364009 · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Compute & fine-tuning and 1 more - S005 EnerVerse-AC: Envisioning Embodied Environments with Action Condition (arXiv:2505.09723v1)
Describes EVAC's multi-level action conditioning, ray map encoding for dynamic multi-view image generation, and failure-trajectory training for data augmentation and offline robotic policy evaluation.Revision, date & where cited
arXiv · arxiv.org · arXiv abstract 2505.09723v1 (2025-05-14) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Setup & usage and 3 more
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from GitHub commit fffbdc364009, HF revision ca05e55b9602, and arXiv:2505.09723v1.