General Navigation Models (GNM, ViNT & NoMaD)
Robotics modelMIT-licensed cross-embodiment visual navigation policy family (GNM, ViNT, and NoMaD) from Berkeley AI Research combining EfficientNet-B0 + Transformer encoders with goal-masked 1D U-Net diffusion for zero-shot topological navigation and exploration on mobile robots, with four PyTorch checkpoints hosted on Google Drive. [1] [2] [3] [4] [5] [6]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Mobile robotics researchers; ROS Noetic / LoCoBot / TurtleBot2 practitioners; Edge-compute visual navigation developers [1]
- Parameters
- Not explicitly stated as a scalar in README or YAML configs (EfficientNet-B0 encoder + 4-layer 4-head Transformer [1] [4]
- Embodiments
- LoCoBot / TurtleBot2 (reference deployment), Clearpath Jackal, Unitree A1, … [1]
- Source rights
- Some licenses are unresolved. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- Ecosystem
- ROS 1, NVIDIA Jetson
- License
- MIT · license not fully assessed for model weights, datasets · Details by part
Overview
robodhruv/visualnav-transformer is Berkeley AI Research's unified PyTorch training and ROS deployment repository for three cross-embodiment visual navigation policies: GNM (ICRA 2023), ViNT (CoRL 2023), and NoMaD (2023). Together they support zero-shot goal-conditioned topological navigation across diverse mobile robots using only a forward-facing RGB camera, while NoMaD adds goal-masked diffusion to unify goal-directed navigation and goal-agnostic exploration in a single model. [1] [5] [6]
Model card summary
- Architecture
Three-model visual navigation family: GNM (obs_encoding_size: 1024, goal_encoding_size: 1024, context_size: 5, image_size: 85x64, len_traj_pred: 5, learn_angle: True per gnm.yaml), ViNT (EfficientNet-B0 encoder + 4-layer 4-head MHA Transformer with context_size: 5, image_size: 85x64, predicting 5 waypoints with angle and distance), and NoMaD (goal-masked ViNT encoder with encoding_size: 256, context_size: 3, image_size: 96x96, goal_mask_prob: 0.5 paired with a 1D U-Net diffusion decoder down_dims: [64, 128, 256], 10 diffusion steps, predicting 8 waypoints for unified navigation and exploration).
- Base model
EfficientNet-B0 (ImageNet-pretrained observation encoder in ViNT and NoMaD)
Verified in train/config/vint.yaml and train/config/nomad.yaml.
- Parameter count
Not explicitly stated as a scalar in README or YAML configs (EfficientNet-B0 encoder + 4-layer 4-head Transformer; NoMaD adds 1D U-Net [64, 128, 256])
Exact parameter counts and checkpoint file sizes are not printed in the README or Google Drive link.
- Input modalities
- rgb
Forward-facing RGB image history (context_size 3 or 5) plus an optional RGB goal image from a topological map (masked during NoMaD exploration).
- Output modalities
- waypoints
Normalized 2D future trajectory waypoints (len_traj_pred: 5 with yaw in ViNT; len_traj_pred: 8 in NoMaD) plus temporal distance, converted to linear/angular velocities by a ROS PD controller.
- Action space
Normalized 2D future waypoint trajectories (5 waypoints + yaw in ViNT; 8 waypoints via 10-step DDPM/DDIM diffusion in NoMaD) tracked by a ROS PD velocity controller (default max_v: 0.2 m/s, max_w: 0.4 rad/s).
- Control frequency
4 Hz configured observation rate (frame_rate: 4 Hz in deployment/config/robot.yaml; not a measured inference benchmark)
- Trained / supported embodiments
- LoCoBot / TurtleBot2 (reference deployment)
- Clearpath Jackal
- Unitree A1
- DJI Tello
- Vizbot
- CARLA simulator
- Training datasets
- RECON (public)
- TartanDrive (public)
- SCAND (public)
- GoStanford2 Modified (public)
- SACSoN / HuRoN (public)
- Cory Hall and unreleased private navigation datasets (e.g., Seattle, per README and configs)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| nomad.pth (NoMaD goal-masked diffusion policy) | pytorch | Not stated | open | dca79815b704e5aa9c6bdc3082351f9e3b2848c2 (Google Drive release; unversioned folder) |
| vint.pth (ViNT visual navigation transformer) | pytorch | Not stated | open | dca79815b704e5aa9c6bdc3082351f9e3b2848c2 (Google Drive release; unversioned folder) |
| vint_late_fusion.pth (ViNT late-fusion variant) | pytorch | Not stated | open | dca79815b704e5aa9c6bdc3082351f9e3b2848c2 (Google Drive release; unversioned folder) |
| gnm_large.pth (GNM general navigation model) | pytorch | Not stated | open | dca79815b704e5aa9c6bdc3082351f9e3b2848c2 (Google Drive release; unversioned folder) |
All four checkpoint filenames verified in deployment/config/models.yaml; hosted in the official Google Drive folder without per-file content hashes or published byte sizes.
Is this for you?
- Use cases
- Zero-shot goal-conditioned topological visual navigation on wheeled and legged robots (GNM, ViNT, NoMaD)
- Goal-agnostic exploration of novel environments using goal-masked diffusion (NoMaD)
- Fine-tuning on custom rosbag or HDF5 robot navigation trajectories
- Known limitations
- Pretrained .pth checkpoints are hosted in an unversioned Google Drive folder rather than a content-addressed model hub, and lack a separate weights license file.
- Training mixture includes unreleased/private datasets (e.g., Seattle) alongside five publicly available navigation datasets.
- Deployment stack targets legacy ROS Noetic on Ubuntu 20.04 and requires external installation of real-stanford/diffusion_policy.
- Safety and limitations
- Relies on a forward-facing wide-angle RGB camera without LiDAR or 3D obstacle sensors; collision avoidance depends on learned visual affordances and a joystick deadman switch during deployment.
- Training incorporates unreleased private datasets alongside public datasets, limiting full from-scratch pretraining reproduction.
- Google Drive checkpoint distribution lacks immutable version hashes and a separate weights license file.
Compute & fine-tuning
- Inference compute
- Documented hardware: NVIDIA Jetson Orin Nano (onboard LoCoBot deployment)
- Precision:
fp32 - Edge / consumer capable: Yes
README documents real-time onboard deployment on LoCoBot with NVIDIA Jetson Orin Nano at 4 Hz.
- Fine-tuning
- Supported: Yes
- Methods: Checkpoint-initialized fine-tuning via load_run in train/config/*.yaml on custom rosbag/HDF5 trajectories
- Recipe: Upstream fine-tuning documentation
- Host computer
Training assumes Ubuntu 18.04/20.04 with CUDA 10+ GPU; edge deployment is documented on NVIDIA Jetson Orin Nano aboard a LoCoBot.
- Operating system
- Linux (Ubuntu 18.04 / 20.04 tested)
- Dependencies
- Python 3.7+
- PyTorch (CUDA 10+ for training)
- real-stanford/diffusion_policy
- ROS Noetic (ros-noetic-usb-cam, ros-noetic-joy, kobuki) for deployment
- tmux (recommended for deployment scripts)
- Skills
- Python
- PyTorch
- ROS Noetic (rosbag, usb_cam, joy)
- Mobile robot kinematics / PD control
Inferred from ROS Noetic deployment scripts and conda training setup.
[1] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [1]
- Create the conda environment from train/train_environment.yml (or deployment/deployment_environment.yaml) and install pip install -e train/
- Clone and install real-stanford/diffusion_policy in editable mode
- Download pretrained .pth checkpoints (nomad.pth, vint.pth, vint_late_fusion.pth, gnm_large.pth) from the linked Google Drive folder into deployment/model_weights/
- Record a ROS bag to build a topological map (record_bag.sh, create_topomap.sh) and deploy via navigate.sh or exploration.sh
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| NoMaD real-world mobile robot navigation and exploration in unseen environments (arXiv:2310.07896v1) | Higher navigation performance and lower collision rates than five alternative baselines with smaller model size (exact tables in arXiv:2310.07896v1) | Upstream-reported real-world mobile robot evaluation; exact per-environment success tables not independently verified by OSR. |
| ViNT cross-embodiment visual navigation and kilometer-scale navigation (arXiv:2306.14846v2) | Positive cross-embodiment transfer outperforming single-dataset specialists across hundreds of hours of navigation data (exact tables in arXiv:2306.14846v2) | Upstream-reported paper claims; not independently verified by OSR. |
Project notes
GNM, ViNT, and NoMaD model configurations
At the pinned commit (dca79815b704), train/config/gnm.yaml configures model_type: gnm with obs_encoding_size: 1024, goal_encoding_size: 1024, context_size: 5, image_size: [85, 64], len_traj_pred: 5, and learn_angle: True (note that the inspected YAML and README do not detail GNM's internal backbone layers beyond these configuration parameters). train/config/vint.yaml pairs an ImageNet-pretrained efficientnet-b0 encoder (obs_encoding_size: 512, image_size: [85, 64], context_size: 5) with a 4-layer, 4-head Transformer to predict 5 normalized future waypoints with yaw (len_traj_pred: 5, learn_angle: True) and temporal distance. train/config/nomad.yaml uses a 256-D EfficientNet-B0 + 4-layer Transformer encoder (image_size: [96, 96], context_size: 3, goal_mask_prob: 0.5) feeding a 1D U-Net diffusion decoder (down_dims: [64, 128, 256], num_diffusion_iters: 10, len_traj_pred: 8) that depends on real-stanford/diffusion_policy. Exact parameter counts and checkpoint file sizes are not printed in the README or YAML configs. [1] [4]
Onboard Jetson deployment workflow and upstream-reported benchmarks
The repository ships a ROS Noetic deployment pipeline tested on a LoCoBot with an NVIDIA Jetson Orin Nano, where deployment/config/robot.yaml configures a frame_rate observation rate of 4 Hz (a configuration setting rather than a measured inference throughput benchmark) and pd_controller.py converts /waypoint outputs to /cmd_vel_mux/input/navi velocity commands alongside a joystick deadman switch. To deploy, the README directs users to activate the vint_deployment conda environment, place the chosen .pth checkpoint in deployment/model_weights/, record a demonstration ROS bag (record_bag.sh) and subsample it into a topological image map (create_topomap.sh), and then launch goal-conditioned navigation via navigate.sh; for goal-agnostic exploration without a target goal, only NoMaD is supported via exploration.sh / explore.sh. Upstream notes independent deployments on Clearpath Jackal, Unitree A1, DJI Tello, TurtleBot2, Vizbot, and CARLA. In arXiv:2310.07896v1 and arXiv:2306.14846v2, the authors report positive cross-embodiment transfer for ViNT over single-dataset specialists and improved navigation performance with lower collision rates for NoMaD against five baselines on real mobile robots. All reported results are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [3] [4] [5] [6]
Open MIT code, unversioned Drive weights, and partial public data
The repository code and documentation are released under the MIT License (LICENSE at dca79815b704). However, the four official PyTorch checkpoints (nomad.pth, vint.pth, vint_late_fusion.pth, and gnm_large.pth, mapped in deployment/config/models.yaml) are distributed via an unversioned Google Drive folder without a separate weights license declaration, so checkpoint license coverage remains unresolved in the supplied evidence (public download access alone does not establish reuse terms). Pretraining combines five public datasets (RECON, TartanDrive, SCAND, GoStanford2, SACSoN/HuRoN)—each requiring separate review of its upstream terms—with unreleased private datasets such as Seattle, preventing full from-scratch reproduction from the publicly released data alone (upstream directs readers to contact the respective authors for access to unreleased data). [1] [2] [3] [4]
What we checked
None. Prepared from public upstream sources without hands-on training, inference, or robot execution.
Updates
- Upstream last updated
- Sep 15, 2024 [1]
- Latest release
- Not established
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- robodhruv/visualnav-transformer@dca79815b704e5aa9c6bdc3082351f9e3b2848c2 · software repository
Separate license matrix
MIT-licensed training and ROS deployment codebase with four publicly downloadable Google Drive checkpoints, classified as partial because checkpoints lack a separate license declaration on Google Drive and pretraining includes unreleased private datasets.
| Layer | Availability / license |
|---|---|
| Code | available · MIT [1] [2] |
| Model weights | available · License not fully assessed Four .pth checkpoints (nomad.pth, vint.pth, vint_late_fusion.pth, gnm_large.pth) are publicly downloadable via Google Drive without a separate weights license file. [1] [3] |
| Training datasets | available · License not fully assessed Partial public availability: five datasets (RECON, TartanDrive, SCAND, GoStanford2, SACSoN/HuRoN) are public with separate upstream licenses, while additional training datasets (e.g., Seattle) remain unreleased. [1] [4] |
| Documentation | available · MIT [1] [2] |
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 robodhruv/visualnav-transformer GitHub repository and README.md
Documents GNM, ViNT, and NoMaD training, Google Drive checkpoint folder, public vs unreleased training datasets, and LoCoBot + Jetson Orin Nano ROS Noetic deployment.Revision, date & where cited
Berkeley AI Research / Dhruv Shah (GitHub) · github.com · revision
dca7981· README.md at commit dca79815b704 (2024-09-15) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 7 more - S002 robodhruv/visualnav-transformer LICENSE
MIT License, Copyright (c) 2023 Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kyle Stachowicz, Kevin Black, Noriaki Hirose, Sergey Levine.Revision, date & where cited
Dhruv Shah et al. (GitHub) · github.com · revision
dca7981· LICENSE at commit dca79815b704 · Accessed Nov 21, 2026
Used for: Open MIT code, unversioned Drive weights, and partial public data, Separate license matrix - S003 Deployment checkpoint registry (deployment/config/models.yaml)
Maps four official checkpoint files: vint.pth, vint_late_fusion.pth, gnm_large.pth, and nomad.pth.Revision, date & where cited
Berkeley AI Research (GitHub) · github.com · revision
dca7981· deployment/config/models.yaml at commit dca79815b704 · Accessed Nov 21, 2026
Used for: Model card summary, Checkpoints, Setup & usage, Onboard Jetson deployment workflow and upstream-reported benchmarks and 2 more - S004 Model and robot configuration files (nomad.yaml, vint.yaml, robot.yaml)
Verifies EfficientNet-B0 + 4-layer 4-head MHA Transformer, NoMaD 1D U-Net [64, 128, 256] with 10 diffusion steps and 8 waypoints, ViNT 5 waypoints with angle, 4 Hz frame_rate, and private dataset placeholders (seattle).Revision, date & where cited
Berkeley AI Research (GitHub) · github.com · revision
dca7981· train/config/nomad.yaml, vint.yaml, and deployment/config/robot.yaml at commit dca79815b704 · Accessed Nov 21, 2026
Used for: Model card summary, Is this for you?, GNM, ViNT, and NoMaD model configurations, Onboard Jetson deployment workflow and upstream-reported benchmarks and 2 more - S005 NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration
Describes unified goal-masked diffusion policy for goal-directed navigation and goal-agnostic exploration on real mobile robots.Revision, date & where cited
arXiv (2310.07896v1) · arxiv.org · arXiv abstract 2310.07896v1 · Accessed Nov 21, 2026
Used for: Overview, Is this for you?, Upstream-reported benchmarks, Onboard Jetson deployment workflow and upstream-reported benchmarks - S006 ViNT: A Foundation Model for Visual Navigation (CoRL 2023)
Describes ViNT cross-embodiment navigation foundation model trained on hundreds of hours of multi-robot navigation data.Revision, date & where cited
arXiv (2306.14846v2) · arxiv.org · arXiv abstract 2306.14846v2 · Accessed Nov 21, 2026
Used for: Overview, Upstream-reported benchmarks, Onboard Jetson deployment workflow and upstream-reported benchmarks
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile drafted from robodhruv/visualnav-transformer@dca79815b704.