Open Source RoboticsRobots, software & learning

DINO-WM

Robotics model

Latent visual world model from NYU and Meta AI that predicts frozen DINOv2 ViT-S/14 spatial patch features with a 6-layer ViT dynamics predictor for zero-shot CEM/GD planning from offline trajectories; MIT code with OSF datasets and 3 environment checkpoints. [1] [2] [3] [5]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Model-based RL and world-model researchers; Zero-shot visual planning researchers; Academic robotics simulation practitioners [2] [5]
Parameters
Not explicitly stated upstream (frozen DINOv2 ViT-S/14 encoder + 6-layer 16-head ViT predictor + optional VQ-VAE decoder [3] [4]
Embodiments
Simulated PointMaze agent (MuJoCo), Simulated PushT planar pushing agent, Simulated Wall navigation environment, … [2] [5]
Source rights
Some licenses are unresolved. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware
Ecosystem
MuJoCo
License
MIT · license not fully assessed for model weights, datasets · Details by part

Requirements · Research details and dates

On this page

Overview

DINO-WM (DINO World Model), developed by researchers at NYU and Meta AI, models visual dynamics in the spatial patch-feature space of a frozen DINOv2 ViT-S/14 (dinov2_vits14) encoder rather than reconstructing raw pixels. A 6-layer, 16-head ViT dynamics predictor (ViTPredictor, mlp_dim=2048) predicts future patch tokens conditioned on proprioception and action embeddings, with an optional VQ-VAE decoder (quantize=False) for visual inspection. At test time, DINO-WM plans zero-shot toward observational goal images by optimizing action sequences via Cross-Entropy Method (CEM) or gradient descent directly in latent feature space. [1] [3] [5]

Model card summary

Architecture

Latent visual world model (VWorldModel) combining a frozen DINOv2 ViT-S/14 (dinov2_vits14) spatial patch-token encoder, proprioception and action encoders, a 6-layer 16-head ViT dynamics predictor (mlp_dim=2048) predicting future patch features, and an optional VQ-VAE decoder (quantize=False) for visualization; zero-shot planning optimizes action sequences via CEM or gradient descent.

[3] [5]

Base model

DINOv2 ViT-S/14 (dinov2_vits14, frozen visual encoder)

[3]

Parameter count

Not explicitly stated upstream (frozen DINOv2 ViT-S/14 encoder + 6-layer 16-head ViT predictor + optional VQ-VAE decoder; 0.95 GB zipped bundle of 3 environment checkpoints on OSF)

[3] [4]

Input modalities
rgbproprioception

224x224 RGB observation frames, proprioceptive state, and continuous control actions (normalized).

[3] [5]

Output modalities
video-frames

Predicts future DINOv2 spatial patch features for CEM/GD action-sequence planning, with an optional VQ-VAE decoder for RGB frame visualization.

[3] [5]

Action space

Environment-specific continuous control actions (embedded via action_encoder with action_emb_dim=10 and frameskip=5 in default config) optimized at test time by CEM or gradient descent in latent DINOv2 feature space.

[2] [3] [5]

Control frequency

Not specified (uses discrete simulation steps with configurable frameskip, default frameskip=5 and num_hist=3)

[3]

Trained / supported embodiments
  • Simulated PointMaze agent (MuJoCo)
  • Simulated PushT planar pushing agent
  • Simulated Wall navigation environment
  • Simulated deformable granular and rope manipulation (PyFleX / SoftGym)

[2] [5]

Training datasets
  • DINO-WM offline trajectory datasets on OSF (bmw48): point_maze, pusht_noise, wall_single, and deformable (granular, rope)

[2] [4]

Interfaces
  • Hydra Python CLI (train.py and plan.py)
  • YAML configuration files (conf/train.yaml, conf/plan_*.yaml)

[2] [3]

Checkpoints

CheckpointFormatSizeAccessPinned revision
DINO-WM pre-trained checkpoints bundle (PointMaze, PushT, Wall — outputs.zip on OSF)pytorch0.95 GBopen2025-03-24

Verified via OSF API: checkpoints/outputs.zip is 953,204,628 bytes (~0.95 decimal GB), modified 2025-03-24T15:45:37Z; contains PointMaze, PushT, and Wall checkpoints.

[2] [4]

Is this for you?

Use cases
  • Zero-shot test-time visual planning (CEM and gradient descent) toward observational goal images
  • Offline world-model training on pre-collected trajectories (PointMaze, PushT, Wall, Deformable granular/rope)
  • Studying latent patch-feature dynamics versus pixel-reconstruction world models

[2] [3] [5]

Known limitations
  • Pre-trained checkpoints uploaded on OSF cover only 3 simulated tasks (PointMaze, PushT, Wall) rather than all 6 paper environments
  • OSF data and checkpoint archive (outputs.zip) does not include an explicit separate weights/data license file (GitHub code is MIT)
  • Relies on legacy MuJoCo 2.10 (mujoco-py) and optional Docker-compiled PyFleX (xingyu/softgym) for deformable environments
  • Evaluated in simulated planning benchmarks in the reviewed materials; physical robot hardware deployment is not documented in the repository

[1] [2] [4] [5]

Safety and limitations
  • Demonstrated on simulated planning benchmarks (MuJoCo/PyFleX) in the reviewed repository; does not include real-robot safety interlocks or hardware deployment validation.
  • Only 3 of the 6 paper environments (PointMaze, PushT, Wall) have pre-trained checkpoints uploaded to OSF.
  • OSF checkpoint and dataset bundle lacks an explicit separate license notice (only the GitHub repository carries an MIT LICENSE).
  • Depends on legacy MuJoCo 2.10 and optional Docker-compiled PyFleX, increasing environment setup friction.

[1] [2] [4]

Compute & fine-tuning

Inference compute
  • Documented hardware: 1x NVIDIA H100 (configured in conf/train.yaml SLURM launcher)
  • Precision: fp32
  • Edge / consumer capable: No

conf/train.yaml specifies gres: gpu:h100:1 for training; exact minimum VRAM and planning latency are not specified in the README.

[3] · Editorial assessment

Fine-tuning

Trains predictor, action/proprio encoders, and optional decoder while keeping the DINOv2 encoder frozen by default (train_encoder: False).

[2] [3]

Host computer

Linux x86-64 with NVIDIA GPU; default Hydra SLURM launcher config (conf/train.yaml) specifies 1x H100 GPU, 8 CPUs, and 512 GB host RAM for training, though exact minimum inference/training VRAM is not specified.

[2] [3]

Operating system
  • Linux x86-64

README downloads mujoco210-linux-x86_64.tar.gz and configures Linux LD_LIBRARY_PATH.

[2]

Dependencies
  • Python and Conda (environment.yaml)
  • PyTorch and Hydra
  • DINOv2 (dinov2_vits14)
  • MuJoCo 2.10 (mujoco210-linux-x86_64)
  • PyFleX via xingyu/softgym Docker (optional for deformable tasks)

[2] [3]

Skills
  • Python, Conda, and Hydra configuration
  • MuJoCo 2.10 / mujoco-py Linux library setup
  • Optional Docker and CMake build for PyFleX deformable simulation

Inferred from README installation and Hydra config structure.

[2] [3] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [2] · Model card / documentation [2]

  • Clone gaoyuezhou/dino_wm and create the Conda environment via environment.yaml
  • Install MuJoCo 2.10 into ~/.mujoco and export LD_LIBRARY_PATH (plus optional Docker PyFleX build for deformable tasks)
  • Download offline datasets and/or pre-trained checkpoints (outputs.zip) from OSF (bmw48) and set DATASET_DIR and ckpt_base_path
  • Train with python train.py --config-name train.yaml or plan with python plan.py --config-name plan_<env>.yaml

[2] [4]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
DINO-WM zero-shot visual planning suite (arXiv:2411.04983v2)Upstream abstract reports zero-shot test-time behavioral solutions across six environments (arbitrarily configured mazes, push manipulation with varied object shapes, and multi-particle deformable scenarios) without expert demos, reward models, or pre-learned inverse models.Upstream-reported paper abstract summary; numerical per-task success rates are in the paper body (not excerpted here) and not independently verified by OSR.

[5]

Project notes

Offline datasets, checkpoints, and environment dependencies

DINO-WM publishes offline trajectory datasets (point_maze, pusht_noise, wall_single, and deformable granular/rope) alongside pre-trained checkpoints on OSF (bmw48). Verified via the OSF API, checkpoints/outputs.zip is 953,204,628 bytes (~0.95 decimal GB, last modified 2025-03-24) and covers three environments: PointMaze, PushT, and Wall (three of the six environments evaluated in the paper). The documented setup uses Linux x86-64, Conda, and MuJoCo 2.10 (mujoco-py), plus an optional Docker-compiled PyFleX (xingyu/softgym) build for deformable environments. The default Hydra SLURM training config (conf/train.yaml) requests 1x NVIDIA H100 GPU, though exact minimum inference or training VRAM is not documented upstream. As an editorial starting route, adopters evaluating pre-trained planning can select PointMaze, PushT, or Wall, follow the documented environment setup, set ckpt_base_path, and run plan.py with the matching planning configuration (plan_point_maze.yaml, plan_pusht.yaml, or plan_wall.yaml)—noting that this published checkpoint route does not cover all six environments evaluated in the paper. [2] [3] [4]

Upstream-reported evaluation and practical scope

In arXiv:2411.04983v2, the authors report zero-shot test-time planning across six simulated environments—including arbitrarily configured mazes, push manipulation with varied object shapes, and multi-particle deformable tasks—without expert demonstrations, reward models, or pre-learned inverse models. These results are upstream-reported and scoped to simulated benchmarks in the reviewed repository; Open Source Robotics has not run training, fine-tuning, or inference on DINO-WM, and physical robot deployment is not documented in the codebase. [2] [5]

What we checked

None. Prepared from public upstream sources without hands-on training, fine-tuning, or inference.

Updates

Upstream last updated
Mar 24, 2025 [1]
Latest release
Mar 24, 2025 [4]
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
gaoyuezhou/dino_wm@0a9492fa12044b852ae9e001cc74604b79c8bb0c; OSF bmw48 checkpoints/outputs.zip (modified 2025-03-24) · software repository

Separate license matrix

Influential MIT-licensed latent visual world model with public offline datasets and checkpoints on OSF; marked include_partial because OSF checkpoints and datasets lack an explicit separate license notice and only 3 of 6 paper environments have uploaded checkpoints.

LayerAvailability / license
Codeavailable · MIT
Verified MIT in root LICENSE file. [1]
Model weightsavailable · License not fully assessed
Openly downloadable outputs.zip (0.95 GB) on OSF for PointMaze, PushT, and Wall; no separate weights license file is published in the OSF folder (code repository is MIT). [2] [4]
Training datasetsavailable · License not fully assessed
Offline datasets (point_maze, pusht_noise, wall_single, deformable) are publicly hosted on OSF project bmw48 without a separate dataset license notice. [2] [4]
Documentationavailable · MIT
[1] [2]

Licensing and maintenance

The supplied materials establish MIT licensing (LICENSE) for the GitHub repository (gaoyuezhou/dino_wm, HEAD 0a9492fa1204, 2025-03-24), which has had no commits since March 2025. However, the supplied materials do not establish separate license terms for the OSF checkpoints (outputs.zip) or offline datasets; readers should clarify artifact licensing with the maintainers before relying on the code repository's MIT license for checkpoint or dataset reuse. [1] [4]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from GitHub commit 0a9492fa1204, OSF bmw48 API metadata, and arXiv:2411.04983v2.