DINO-WM
Robotics modelLatent visual world model from NYU and Meta AI that predicts frozen DINOv2 ViT-S/14 spatial patch features with a 6-layer ViT dynamics predictor for zero-shot CEM/GD planning from offline trajectories; MIT code with OSF datasets and 3 environment checkpoints. [1] [2] [3] [5]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Model-based RL and world-model researchers; Zero-shot visual planning researchers; Academic robotics simulation practitioners [2] [5]
- Parameters
- Not explicitly stated upstream (frozen DINOv2 ViT-S/14 encoder + 6-layer 16-head ViT predictor + optional VQ-VAE decoder [3] [4]
- Embodiments
- Simulated PointMaze agent (MuJoCo), Simulated PushT planar pushing agent, Simulated Wall navigation environment, … [2] [5]
- Source rights
- Some licenses are unresolved. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- Ecosystem
- MuJoCo
- License
- MIT · license not fully assessed for model weights, datasets · Details by part
Overview
DINO-WM (DINO World Model), developed by researchers at NYU and Meta AI, models visual dynamics in the spatial patch-feature space of a frozen DINOv2 ViT-S/14 (dinov2_vits14) encoder rather than reconstructing raw pixels. A 6-layer, 16-head ViT dynamics predictor (ViTPredictor, mlp_dim=2048) predicts future patch tokens conditioned on proprioception and action embeddings, with an optional VQ-VAE decoder (quantize=False) for visual inspection. At test time, DINO-WM plans zero-shot toward observational goal images by optimizing action sequences via Cross-Entropy Method (CEM) or gradient descent directly in latent feature space. [1] [3] [5]
Model card summary
- Architecture
Latent visual world model (VWorldModel) combining a frozen DINOv2 ViT-S/14 (dinov2_vits14) spatial patch-token encoder, proprioception and action encoders, a 6-layer 16-head ViT dynamics predictor (mlp_dim=2048) predicting future patch features, and an optional VQ-VAE decoder (quantize=False) for visualization; zero-shot planning optimizes action sequences via CEM or gradient descent.
- Base model
DINOv2 ViT-S/14 (dinov2_vits14, frozen visual encoder)
- Parameter count
Not explicitly stated upstream (frozen DINOv2 ViT-S/14 encoder + 6-layer 16-head ViT predictor + optional VQ-VAE decoder; 0.95 GB zipped bundle of 3 environment checkpoints on OSF)
- Input modalities
- rgbproprioception
224x224 RGB observation frames, proprioceptive state, and continuous control actions (normalized).
- Output modalities
- video-frames
Predicts future DINOv2 spatial patch features for CEM/GD action-sequence planning, with an optional VQ-VAE decoder for RGB frame visualization.
- Action space
Environment-specific continuous control actions (embedded via action_encoder with action_emb_dim=10 and frameskip=5 in default config) optimized at test time by CEM or gradient descent in latent DINOv2 feature space.
- Control frequency
Not specified (uses discrete simulation steps with configurable frameskip, default frameskip=5 and num_hist=3)
- Trained / supported embodiments
- Simulated PointMaze agent (MuJoCo)
- Simulated PushT planar pushing agent
- Simulated Wall navigation environment
- Simulated deformable granular and rope manipulation (PyFleX / SoftGym)
- Training datasets
- DINO-WM offline trajectory datasets on OSF (bmw48): point_maze, pusht_noise, wall_single, and deformable (granular, rope)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| DINO-WM pre-trained checkpoints bundle (PointMaze, PushT, Wall — outputs.zip on OSF) | pytorch | 0.95 GB | open | 2025-03-24 |
Verified via OSF API: checkpoints/outputs.zip is 953,204,628 bytes (~0.95 decimal GB), modified 2025-03-24T15:45:37Z; contains PointMaze, PushT, and Wall checkpoints.
Is this for you?
- Use cases
- Zero-shot test-time visual planning (CEM and gradient descent) toward observational goal images
- Offline world-model training on pre-collected trajectories (PointMaze, PushT, Wall, Deformable granular/rope)
- Studying latent patch-feature dynamics versus pixel-reconstruction world models
- Known limitations
- Pre-trained checkpoints uploaded on OSF cover only 3 simulated tasks (PointMaze, PushT, Wall) rather than all 6 paper environments
- OSF data and checkpoint archive (outputs.zip) does not include an explicit separate weights/data license file (GitHub code is MIT)
- Relies on legacy MuJoCo 2.10 (mujoco-py) and optional Docker-compiled PyFleX (xingyu/softgym) for deformable environments
- Evaluated in simulated planning benchmarks in the reviewed materials; physical robot hardware deployment is not documented in the repository
- Safety and limitations
- Demonstrated on simulated planning benchmarks (MuJoCo/PyFleX) in the reviewed repository; does not include real-robot safety interlocks or hardware deployment validation.
- Only 3 of the 6 paper environments (PointMaze, PushT, Wall) have pre-trained checkpoints uploaded to OSF.
- OSF checkpoint and dataset bundle lacks an explicit separate license notice (only the GitHub repository carries an MIT LICENSE).
- Depends on legacy MuJoCo 2.10 and optional Docker-compiled PyFleX, increasing environment setup friction.
Compute & fine-tuning
- Inference compute
- Documented hardware: 1x NVIDIA H100 (configured in conf/train.yaml SLURM launcher)
- Precision:
fp32 - Edge / consumer capable: No
conf/train.yaml specifies gres: gpu:h100:1 for training; exact minimum VRAM and planning latency are not specified in the README.
[3] · Editorial assessment
- Fine-tuning
- Supported: Yes
- Methods: Full world-model training on offline trajectory datasets (train.py)
- Recipe: Upstream fine-tuning documentation
Trains predictor, action/proprio encoders, and optional decoder while keeping the DINOv2 encoder frozen by default (train_encoder: False).
- Host computer
Linux x86-64 with NVIDIA GPU; default Hydra SLURM launcher config (conf/train.yaml) specifies 1x H100 GPU, 8 CPUs, and 512 GB host RAM for training, though exact minimum inference/training VRAM is not specified.
- Operating system
- Linux x86-64
README downloads mujoco210-linux-x86_64.tar.gz and configures Linux LD_LIBRARY_PATH.
- Dependencies
- Python and Conda (environment.yaml)
- PyTorch and Hydra
- DINOv2 (dinov2_vits14)
- MuJoCo 2.10 (mujoco210-linux-x86_64)
- PyFleX via xingyu/softgym Docker (optional for deformable tasks)
Setup & usage
Open the upstream getting-started guide [2] · Model card / documentation [2]
- Clone gaoyuezhou/dino_wm and create the Conda environment via environment.yaml
- Install MuJoCo 2.10 into ~/.mujoco and export LD_LIBRARY_PATH (plus optional Docker PyFleX build for deformable tasks)
- Download offline datasets and/or pre-trained checkpoints (outputs.zip) from OSF (bmw48) and set DATASET_DIR and ckpt_base_path
- Train with python train.py --config-name train.yaml or plan with python plan.py --config-name plan_<env>.yaml
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| DINO-WM zero-shot visual planning suite (arXiv:2411.04983v2) | Upstream abstract reports zero-shot test-time behavioral solutions across six environments (arbitrarily configured mazes, push manipulation with varied object shapes, and multi-particle deformable scenarios) without expert demos, reward models, or pre-learned inverse models. | Upstream-reported paper abstract summary; numerical per-task success rates are in the paper body (not excerpted here) and not independently verified by OSR. |
Project notes
Offline datasets, checkpoints, and environment dependencies
DINO-WM publishes offline trajectory datasets (point_maze, pusht_noise, wall_single, and deformable granular/rope) alongside pre-trained checkpoints on OSF (bmw48). Verified via the OSF API, checkpoints/outputs.zip is 953,204,628 bytes (~0.95 decimal GB, last modified 2025-03-24) and covers three environments: PointMaze, PushT, and Wall (three of the six environments evaluated in the paper). The documented setup uses Linux x86-64, Conda, and MuJoCo 2.10 (mujoco-py), plus an optional Docker-compiled PyFleX (xingyu/softgym) build for deformable environments. The default Hydra SLURM training config (conf/train.yaml) requests 1x NVIDIA H100 GPU, though exact minimum inference or training VRAM is not documented upstream. As an editorial starting route, adopters evaluating pre-trained planning can select PointMaze, PushT, or Wall, follow the documented environment setup, set ckpt_base_path, and run plan.py with the matching planning configuration (plan_point_maze.yaml, plan_pusht.yaml, or plan_wall.yaml)—noting that this published checkpoint route does not cover all six environments evaluated in the paper. [2] [3] [4]
Upstream-reported evaluation and practical scope
In arXiv:2411.04983v2, the authors report zero-shot test-time planning across six simulated environments—including arbitrarily configured mazes, push manipulation with varied object shapes, and multi-particle deformable tasks—without expert demonstrations, reward models, or pre-learned inverse models. These results are upstream-reported and scoped to simulated benchmarks in the reviewed repository; Open Source Robotics has not run training, fine-tuning, or inference on DINO-WM, and physical robot deployment is not documented in the codebase. [2] [5]
What we checked
None. Prepared from public upstream sources without hands-on training, fine-tuning, or inference.
Updates
- Upstream last updated
- Mar 24, 2025 [1]
- Latest release
- Mar 24, 2025 [4]
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- gaoyuezhou/dino_wm@0a9492fa12044b852ae9e001cc74604b79c8bb0c; OSF bmw48 checkpoints/outputs.zip (modified 2025-03-24) · software repository
Separate license matrix
Influential MIT-licensed latent visual world model with public offline datasets and checkpoints on OSF; marked include_partial because OSF checkpoints and datasets lack an explicit separate license notice and only 3 of 6 paper environments have uploaded checkpoints.
| Layer | Availability / license |
|---|---|
| Code | available · MIT Verified MIT in root LICENSE file. [1] |
| Model weights | available · License not fully assessed Openly downloadable outputs.zip (0.95 GB) on OSF for PointMaze, PushT, and Wall; no separate weights license file is published in the OSF folder (code repository is MIT). [2] [4] |
| Training datasets | available · License not fully assessed Offline datasets (point_maze, pusht_noise, wall_single, deformable) are publicly hosted on OSF project bmw48 without a separate dataset license notice. [2] [4] |
| Documentation | available · MIT [1] [2] |
Licensing and maintenance
The supplied materials establish MIT licensing (LICENSE) for the GitHub repository (gaoyuezhou/dino_wm, HEAD 0a9492fa1204, 2025-03-24), which has had no commits since March 2025. However, the supplied materials do not establish separate license terms for the OSF checkpoints (outputs.zip) or offline datasets; readers should clarify artifact licensing with the maintainers before relying on the code repository's MIT license for checkpoint or dataset reuse. [1] [4]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 gaoyuezhou/dino_wm GitHub metadata, HEAD commit, and LICENSE
Default branch main HEAD is 0a9492fa12044b852ae9e001cc74604b79c8bb0c (2025-03-24T19:46:34Z); archived=false; root LICENSE is MIT.Revision, date & where cited
gaoyuezhou (GitHub) · github.com · revision
0a9492f· GitHub API repos/gaoyuezhou/dino_wm & root LICENSE at 0a9492fa1204 · Accessed Nov 21, 2026
Used for: Overview, Is this for you?, What we checked, Separate license matrix - S002 gaoyuezhou/dino_wm README.md at pinned commit
Documents Conda + MuJoCo 2.10 + optional Docker PyFleX installation, OSF dataset and checkpoint download (PointMaze, PushT, Wall), and train.py / plan.py commands.Revision, date & where cited
gaoyuezhou (GitHub) · github.com · revision
0a9492f· README.md at commit 0a9492fa1204 · Accessed Nov 21, 2026
Used for: Model card summary, Checkpoints, Is this for you?, Compute & fine-tuning and 4 more - S003 DINO-WM Hydra model and training configs (conf/train.yaml, encoder/dino.yaml, predictor/vit.yaml, decoder/vqvae.yaml)
Specifies frozen dinov2_vits14 patch-token encoder, 6-layer 16-head ViTPredictor (mlp_dim=2048), optional VQVAE decoder (quantize=False), 224x224 inputs, frameskip=5, num_hist=3, and CEM/GD planners.Revision, date & where cited
gaoyuezhou (GitHub) · github.com · revision
0a9492f· conf/train.yaml, conf/encoder/dino.yaml, conf/predictor/vit.yaml, conf/decoder/vqvae.yaml · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Compute & fine-tuning and 1 more - S004 OSF bmw48 storage API metadata (checkpoints and datasets)
Confirms public datasets/ folder and checkpoints/outputs.zip (953,204,628 bytes, modified 2025-03-24T15:45:37Z).Revision, date & where cited
OSF · osf.io · OSF API /v2/nodes/bmw48/files/osfstorage/ · Accessed Nov 21, 2026
Used for: Model card summary, Checkpoints, Is this for you?, Setup & usage and 3 more - S005 DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning (arXiv:2411.04983v2)
Describes predicting future DINOv2 patch features from offline trajectories for zero-shot test-time action optimization across six environments without expert demonstrations, reward modeling, or inverse models.Revision, date & where cited
arXiv (NYU / Meta AI) · arxiv.org · arXiv abstract 2411.04983v2 (2025-02-01) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Upstream-reported benchmarks and 1 more
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from GitHub commit 0a9492fa1204, OSF bmw48 API metadata, and arXiv:2411.04983v2.