Open Source RoboticsRobots, software & learning

EnerVerse-AC

Robotics model

Action-conditional latent video diffusion world model (EVAC) from AgiBot for generating 320x512 robotic manipulation rollouts conditioned on robot actions and camera ray maps; strictly noncommercial CC BY-NC-SA 4.0 single-view inference repo and AgiBot-World-only checkpoint. [1] [2] [3] [4]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Robotic manipulation and world-model researchers; AgiBot-World dataset and IROS 2025 challenge participants; Noncommercial embodied policy evaluation researchers [2] [3] [5]
Parameters
Not explicitly stated upstream (6.38 GB DeepSpeed PyTorch checkpoint EnerV_AC_deepspeed_v0.1.pt [2] [3]
Embodiments
AgiBot-World bimanual manipulation robot (single-view head camera in released checkpoint) [2] [4]
Source rights
Some licenses are unresolved. Includes noncommercial terms. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware
License
CC BY-NC-SA 4.0 (stated in README; no root LICENSE file) (noncommercial), CC BY-NC-SA 4.0 (noncommercial) (noncommercial), CC BY-NC-SA 4.0 (noncommercial) · license not fully assessed for datasets · Details by part

Requirements · Research details and dates

On this page

Overview

EnerVerse-AC (EVAC), developed by AgiBot (AgibotTech), is an action-conditional latent video diffusion world model (ACWMLatentDiffusion) built on the LVDM / DynamiCrafter 3D causal UNet architecture with an AutoencoderKL, a frozen OpenCLIP ViT-H/14 image embedder, Resampler action and image projectors, sparse memory, and 6-channel camera ray map encoding. Conditioned on an initial RGB frame, delta robot actions, and camera extrinsics/intrinsics, the released configuration generates 16-frame chunks at 320x512 resolution (15 FPS default save rate) from a single head-camera view. [1] [2] [4] [5]

Model card summary

Architecture

Action-conditional latent video diffusion world model (ACWMLatentDiffusion, built on LVDM/DynamiCrafter 3D causal UNet with AutoencoderKL, frozen OpenCLIP ViT-H/14 image embedder, Resampler action/image projectors, sparse memory, and 6-channel camera ray map encoding) generating 16-frame chunks at 320x512 resolution.

[2] [4] [5]

Base model

LVDM / DynamiCrafter 3D causal UNet architecture + frozen laion/CLIP-ViT-H-14-laion2B-s32B-b79K

[2] [4]

Parameter count

Not explicitly stated upstream (6.38 GB DeepSpeed PyTorch checkpoint EnerV_AC_deepspeed_v0.1.pt; requires separate laion/CLIP-ViT-H-14-laion2B-s32B-b79K weights)

[2] [3]

Input modalities
rgbproprioception

Initial RGB frame, delta action trajectories (action_dim=7 per arm, embedding_dim=14 in Resampler), and camera extrinsics/intrinsics encoded as 6-channel ray maps.

[2] [4]

Output modalities
video-frames

Single-view 320x512 RGB video rollouts (16-frame chunks, save_fps=15 in config).

[2] [4]

Action space

Delta robot action trajectories (cond_stage_key: delta_action; action_dim=7, Resampler embedding_dim=14) paired with camera extrinsics and intrinsics conditioning future video frame generation rather than directly outputting motor commands.

[2] [4]

Control frequency

15 Hz default save rate (save_fps: 15 in configs/agibotworld/config.yaml; offline chunked video generation)

[4]

Trained / supported embodiments
  • AgiBot-World bimanual manipulation robot (single-view head camera in released checkpoint)

[2] [4]

Training datasets
  • AgiBot-World open-source dataset only (released checkpoint excludes commercially restricted datasets and failure trajectories referenced in the paper)

[2] [3]

Interfaces
  • Python CLI (main/generate_video_acwm.py)
  • AgiBot-World preprocessing CLI (tools/prepare_infer_data.py)
  • YAML model configuration (configs/agibotworld/config.yaml)

[2] [4]

Checkpoints

CheckpointFormatSizeAccessPinned revision
agibot-world/EnerVerse-AC (single-view, EnerV_AC_deepspeed_v0.1.pt, CC BY-NC-SA 4.0)pytorch6.38 GBopenca05e55b9602

Verified via HF API: EnerV_AC_deepspeed_v0.1.pt is 6,384,255,240 bytes (~6.38 decimal GB); ungated but strictly noncommercial CC BY-NC-SA 4.0. Requires separate laion/CLIP-ViT-H-14-laion2B-s32B-b79K download.

[2] [3]

Is this for you?

Use cases
  • Single-view action-conditioned robotic manipulation video rollout generation
  • Offline visual evaluation of manipulation policies without physical robots
  • Baseline world model for the AgiBot World Challenge @ IROS 2025

[2] [5]

Known limitations
  • Strictly noncommercial CC BY-NC-SA 4.0 license stated in README and HF model card, and the GitHub repository lacks a root LICENSE file
  • Released checkpoint is single-view only (head camera) and trained exclusively on open-source AgiBot-World without the failure-trajectory data or restricted commercial datasets described in the paper
  • Main repository provides minimal inference code only (training code is linked separately in AgibotTech/AgiBotWorldChallengeIROS2025-WorldModelBaseline)
  • Requires manually downloading laion/CLIP-ViT-H-14-laion2B-s32B-b79K and installing a CUDA 12.1 PyTorch3D wheel alongside PyTorch 2.4.0

[1] [2] [3] [4]

Safety and limitations
  • Generates action-conditioned RGB video rollouts for offline evaluation rather than physical robot control commands; the released checkpoint excludes failure trajectories, which may bias policy evaluation toward successful outcomes.
  • Strictly noncommercial CC BY-NC-SA 4.0 license (stated in README and HF model card, with no root LICENSE file in GitHub).
  • Released checkpoint is single-view head-camera only, differing from the multi-view setup described in the paper.
  • Training code is omitted from the main repository, and minimum GPU VRAM / inference latency are unspecified.

[1] [2] [3] [4]

Compute & fine-tuning

Inference compute
  • Documented hardware: Not specified upstream (tested with CUDA 11.8+, PyTorch 2.4.0, fp16 Lightning precision)
  • Precision: fp16
  • Edge / consumer capable: No

Exact minimum GPU VRAM and generation latency are not stated in the README or config.

[2] [4] · Editorial assessment

Fine-tuning
  • Supported: No

AgibotTech/EnerVerse-AC contains minimal inference code only; README directs users seeking minimal training code to the separate AgibotTech/AgiBotWorldChallengeIROS2025-WorldModelBaseline repository.

[2]

Host computer

NVIDIA CUDA GPU host (tested with CUDA 11.8+, PyTorch 2.4.0, fp16 Lightning precision in config); exact minimum GPU VRAM is not stated upstream.

[2] [4]

Operating system
  • Linux

Inferred from CUDA/PyTorch3D Linux wheel and Conda setup instructions.

[2] · Editorial assessment

Dependencies
  • Python 3.10.4
  • PyTorch 2.4.0 and CUDA 11.8+ (PyTorch3D wheel built on CUDA 12.1)
  • PyTorch Lightning / LVDM codebase dependencies
  • laion/CLIP-ViT-H-14-laion2B-s32B-b79K (separate external download)

[2] [4]

Skills
  • Python 3.10 and Conda / CUDA wheel management
  • PyTorch, PyTorch3D, and YAML configuration
  • Camera extrinsics/intrinsics and AgiBot-World data formatting

Inferred from README setup and tools/prepare_infer_data.py inputs.

[2] [4] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [2] · Model card / documentation [2] [5]

  • Clone AgibotTech/EnerVerse-AC, create a Python 3.10.4 Conda environment, install requirements.txt (PyTorch 2.4.0), and install the prebuilt PyTorch3D wheel (py310_cu121_pyt240)
  • Download EnerV_AC_deepspeed_v0.1.pt from agibot-world/EnerVerse-AC and open_clip_pytorch_model.bin from laion/CLIP-ViT-H-14-laion2B-s32B-b79K, updating paths in configs/agibotworld/config.yaml
  • Prepare start frame, action, extrinsic, and intrinsic files (or use tools/prepare_infer_data.py / examples/examples0)
  • Run python main/generate_video_acwm.py with --ckp_path and --config_path

[2] [3]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
EnerVerse-AC robotic manipulation evaluation (arXiv:2505.09723v1) & HF validation metricsUpstream paper abstract reports high-fidelity action-conditioned video generation for policy testing and data augmentation; HF repo also includes EVAC_validation_set_metrics.csv (not summarized in README).Upstream-reported paper claims apply to the full paper model (which included restricted data and failure trajectories not present in the released AgiBot-World-only single-view checkpoint); not independently verified by OSR.

[2] [3] [5]

Project notes

Scope gap between the paper and the released checkpoint

Adopters should note a material gap between the paper (arXiv:2505.09723v1) and the released weights. While the paper presents multi-view generation and highlights expanding training data with diverse failure trajectories for realistic offline policy evaluation, both the GitHub README and Hugging Face model card state that—due to commercial restrictions on portions of the paper's training data—the released checkpoint (EnerV_AC_deepspeed_v0.1.pt, 6.38 GB) is single-view only, was trained exclusively on the open-source AgiBot-World dataset, and excludes failure trajectory data. As an editorial implication, adopters should not assume that the released checkpoint reproduces the paper's multi-view capabilities or failure-sensitive evaluation performance, and the supplied documentation does not quantify the effect of excluding failure trajectories. Moreover, AgibotTech/EnerVerse-AC provides minimal inference code only; minimal training code is hosted separately in AgibotTech/AgiBotWorldChallengeIROS2025-WorldModelBaseline. [2] [3] [5]

Setup constraints and upstream-reported evaluation

The upstream-tested environment uses Python 3.10.4, CUDA 11.8 (higher CUDA versions noted as expected to work), and PyTorch 2.4.0, while the documented installation recipe installs a prebuilt PyTorch3D wheel built on CUDA 12.1 alongside a separate manual download of laion/CLIP-ViT-H-14-laion2B-s32B-b79K wired into configs/agibotworld/config.yaml. Exact parameter counts, minimum GPU VRAM, and generation latency are not specified in the repository README or config. Although the paper reports high-fidelity action-conditioned video generation for policy testing and the Hugging Face repository includes EVAC_validation_set_metrics.csv, those results are upstream-reported (and paper claims reflect the broader non-public training set); Open Source Robotics has not run training, fine-tuning, or inference on EnerVerse-AC. [2] [3] [4] [5]

What we checked

None. Prepared from public upstream sources without hands-on training, fine-tuning, or inference.

Updates

Upstream last updated
Jul 14, 2025 [1]
Latest release
Jun 13, 2025 [3]
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
AgibotTech/EnerVerse-AC@fffbdc364009ba30080c837b79f125e8c7d2d2d1; HF agibot-world/EnerVerse-AC@ca05e55b9602666796d6591a981c7ec50d27fc58 · software repository

Separate license matrix

Notable embodied action-conditioned video world model trained on AgiBot-World and used as the IROS 2025 challenge baseline; marked include_partial because both code and weights are strictly noncommercial CC BY-NC-SA 4.0 (with no root LICENSE file), only single-view inference without failure trajectories is released, and training code lives in a separate repo.

LayerAvailability / license
Codeavailable · CC BY-NC-SA 4.0 (stated in README; no root LICENSE file)
Minimal single-view inference code only; GitHub API confirms no root LICENSE file exists, while README declares CC BY-NC-SA 4.0. [1] [2]
Model weightsavailable · CC BY-NC-SA 4.0 (noncommercial)
Ungated 6.38 GB single-view checkpoint under noncommercial CC BY-NC-SA 4.0; trained only on open-source AgiBot-World without failure trajectories. [3]
Training datasetsavailable · License not fully assessed
Released checkpoint was trained exclusively on open-source AgiBot-World (external upstream license not separately audited here); commercially restricted datasets and failure trajectories from the paper are not released. [2] [3]
Documentationavailable · CC BY-NC-SA 4.0
[2]

Noncommercial licensing and repository hygiene

Both the GitHub README and the Hugging Face repository declare CC BY-NC-SA 4.0 (strictly noncommercial, share-alike) for all code and weights, even though the GitHub repository (AgibotTech/EnerVerse-AC, HEAD fffbdc364009, 2025-07-14, with no commits since July 2025) does not contain a root LICENSE file. Readers considering commercial use should seek separate permission rather than treating the missing root LICENSE as permissive, and teams requiring a formal license record should request clarification from the maintainer. [1] [2] [3]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from GitHub commit fffbdc364009, HF revision ca05e55b9602, and arXiv:2505.09723v1.