Open Source RoboticsRobots, software & learning

General Navigation Models (GNM, ViNT & NoMaD)

Robotics model

MIT-licensed cross-embodiment visual navigation policy family (GNM, ViNT, and NoMaD) from Berkeley AI Research combining EfficientNet-B0 + Transformer encoders with goal-masked 1D U-Net diffusion for zero-shot topological navigation and exploration on mobile robots, with four PyTorch checkpoints hosted on Google Drive. [1] [2] [3] [4] [5] [6]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Mobile robotics researchers; ROS Noetic / LoCoBot / TurtleBot2 practitioners; Edge-compute visual navigation developers [1]
Parameters
Not explicitly stated as a scalar in README or YAML configs (EfficientNet-B0 encoder + 4-layer 4-head Transformer [1] [4]
Embodiments
LoCoBot / TurtleBot2 (reference deployment), Clearpath Jackal, Unitree A1, … [1]
Source rights
Some licenses are unresolved. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware
License
MIT · license not fully assessed for model weights, datasets · Details by part

Requirements · Research details and dates

On this page

Overview

robodhruv/visualnav-transformer is Berkeley AI Research's unified PyTorch training and ROS deployment repository for three cross-embodiment visual navigation policies: GNM (ICRA 2023), ViNT (CoRL 2023), and NoMaD (2023). Together they support zero-shot goal-conditioned topological navigation across diverse mobile robots using only a forward-facing RGB camera, while NoMaD adds goal-masked diffusion to unify goal-directed navigation and goal-agnostic exploration in a single model. [1] [5] [6]

Model card summary

Architecture

Three-model visual navigation family: GNM (obs_encoding_size: 1024, goal_encoding_size: 1024, context_size: 5, image_size: 85x64, len_traj_pred: 5, learn_angle: True per gnm.yaml), ViNT (EfficientNet-B0 encoder + 4-layer 4-head MHA Transformer with context_size: 5, image_size: 85x64, predicting 5 waypoints with angle and distance), and NoMaD (goal-masked ViNT encoder with encoding_size: 256, context_size: 3, image_size: 96x96, goal_mask_prob: 0.5 paired with a 1D U-Net diffusion decoder down_dims: [64, 128, 256], 10 diffusion steps, predicting 8 waypoints for unified navigation and exploration).

[1] [4]

Base model

EfficientNet-B0 (ImageNet-pretrained observation encoder in ViNT and NoMaD)

Verified in train/config/vint.yaml and train/config/nomad.yaml.

[4]

Parameter count

Not explicitly stated as a scalar in README or YAML configs (EfficientNet-B0 encoder + 4-layer 4-head Transformer; NoMaD adds 1D U-Net [64, 128, 256])

Exact parameter counts and checkpoint file sizes are not printed in the README or Google Drive link.

[1] [4] · Editorial assessment

Input modalities
rgb

Forward-facing RGB image history (context_size 3 or 5) plus an optional RGB goal image from a topological map (masked during NoMaD exploration).

[1] [4]

Output modalities
waypoints

Normalized 2D future trajectory waypoints (len_traj_pred: 5 with yaw in ViNT; len_traj_pred: 8 in NoMaD) plus temporal distance, converted to linear/angular velocities by a ROS PD controller.

[1] [4]

Action space

Normalized 2D future waypoint trajectories (5 waypoints + yaw in ViNT; 8 waypoints via 10-step DDPM/DDIM diffusion in NoMaD) tracked by a ROS PD velocity controller (default max_v: 0.2 m/s, max_w: 0.4 rad/s).

[1] [4]

Control frequency

4 Hz configured observation rate (frame_rate: 4 Hz in deployment/config/robot.yaml; not a measured inference benchmark)

[4]

Trained / supported embodiments
  • LoCoBot / TurtleBot2 (reference deployment)
  • Clearpath Jackal
  • Unitree A1
  • DJI Tello
  • Vizbot
  • CARLA simulator

[1]

Training datasets
  • RECON (public)
  • TartanDrive (public)
  • SCAND (public)
  • GoStanford2 Modified (public)
  • SACSoN / HuRoN (public)
  • Cory Hall and unreleased private navigation datasets (e.g., Seattle, per README and configs)

[1] [4]

Interfaces
  • ROS Noetic topics (/usb_cam/image_raw, /waypoint, /cmd_vel_mux/input/navi)
  • Python training CLI (train/train.py -c <config.yaml>)
  • YAML model and robot configs (models.yaml, robot.yaml)

[1] [3] [4]

Checkpoints

CheckpointFormatSizeAccessPinned revision
nomad.pth (NoMaD goal-masked diffusion policy)pytorchNot statedopendca79815b704e5aa9c6bdc3082351f9e3b2848c2 (Google Drive release; unversioned folder)
vint.pth (ViNT visual navigation transformer)pytorchNot statedopendca79815b704e5aa9c6bdc3082351f9e3b2848c2 (Google Drive release; unversioned folder)
vint_late_fusion.pth (ViNT late-fusion variant)pytorchNot statedopendca79815b704e5aa9c6bdc3082351f9e3b2848c2 (Google Drive release; unversioned folder)
gnm_large.pth (GNM general navigation model)pytorchNot statedopendca79815b704e5aa9c6bdc3082351f9e3b2848c2 (Google Drive release; unversioned folder)

All four checkpoint filenames verified in deployment/config/models.yaml; hosted in the official Google Drive folder without per-file content hashes or published byte sizes.

[1] [3]

Is this for you?

Use cases
  • Zero-shot goal-conditioned topological visual navigation on wheeled and legged robots (GNM, ViNT, NoMaD)
  • Goal-agnostic exploration of novel environments using goal-masked diffusion (NoMaD)
  • Fine-tuning on custom rosbag or HDF5 robot navigation trajectories

[1] [5]

Known limitations
  • Pretrained .pth checkpoints are hosted in an unversioned Google Drive folder rather than a content-addressed model hub, and lack a separate weights license file.
  • Training mixture includes unreleased/private datasets (e.g., Seattle) alongside five publicly available navigation datasets.
  • Deployment stack targets legacy ROS Noetic on Ubuntu 20.04 and requires external installation of real-stanford/diffusion_policy.

[1] [4]

Safety and limitations
  • Relies on a forward-facing wide-angle RGB camera without LiDAR or 3D obstacle sensors; collision avoidance depends on learned visual affordances and a joystick deadman switch during deployment.
  • Training incorporates unreleased private datasets alongside public datasets, limiting full from-scratch pretraining reproduction.
  • Google Drive checkpoint distribution lacks immutable version hashes and a separate weights license file.

[1] [4]

Compute & fine-tuning

Inference compute
  • Documented hardware: NVIDIA Jetson Orin Nano (onboard LoCoBot deployment)
  • Precision: fp32
  • Edge / consumer capable: Yes

README documents real-time onboard deployment on LoCoBot with NVIDIA Jetson Orin Nano at 4 Hz.

[1]

Fine-tuning
  • Supported: Yes
  • Methods: Checkpoint-initialized fine-tuning via load_run in train/config/*.yaml on custom rosbag/HDF5 trajectories
  • Recipe: Upstream fine-tuning documentation

[1]

Host computer

Training assumes Ubuntu 18.04/20.04 with CUDA 10+ GPU; edge deployment is documented on NVIDIA Jetson Orin Nano aboard a LoCoBot.

[1]

Operating system
  • Linux (Ubuntu 18.04 / 20.04 tested)

[1]

Dependencies
  • Python 3.7+
  • PyTorch (CUDA 10+ for training)
  • real-stanford/diffusion_policy
  • ROS Noetic (ros-noetic-usb-cam, ros-noetic-joy, kobuki) for deployment
  • tmux (recommended for deployment scripts)

[1]

Skills
  • Python
  • PyTorch
  • ROS Noetic (rosbag, usb_cam, joy)
  • Mobile robot kinematics / PD control

Inferred from ROS Noetic deployment scripts and conda training setup.

[1] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1] · Model card / documentation [1]

  • Create the conda environment from train/train_environment.yml (or deployment/deployment_environment.yaml) and install pip install -e train/
  • Clone and install real-stanford/diffusion_policy in editable mode
  • Download pretrained .pth checkpoints (nomad.pth, vint.pth, vint_late_fusion.pth, gnm_large.pth) from the linked Google Drive folder into deployment/model_weights/
  • Record a ROS bag to build a topological map (record_bag.sh, create_topomap.sh) and deploy via navigate.sh or exploration.sh

[1] [3]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
NoMaD real-world mobile robot navigation and exploration in unseen environments (arXiv:2310.07896v1)Higher navigation performance and lower collision rates than five alternative baselines with smaller model size (exact tables in arXiv:2310.07896v1)Upstream-reported real-world mobile robot evaluation; exact per-environment success tables not independently verified by OSR.
ViNT cross-embodiment visual navigation and kilometer-scale navigation (arXiv:2306.14846v2)Positive cross-embodiment transfer outperforming single-dataset specialists across hundreds of hours of navigation data (exact tables in arXiv:2306.14846v2)Upstream-reported paper claims; not independently verified by OSR.

[5] [6]

Project notes

GNM, ViNT, and NoMaD model configurations

At the pinned commit (dca79815b704), train/config/gnm.yaml configures model_type: gnm with obs_encoding_size: 1024, goal_encoding_size: 1024, context_size: 5, image_size: [85, 64], len_traj_pred: 5, and learn_angle: True (note that the inspected YAML and README do not detail GNM's internal backbone layers beyond these configuration parameters). train/config/vint.yaml pairs an ImageNet-pretrained efficientnet-b0 encoder (obs_encoding_size: 512, image_size: [85, 64], context_size: 5) with a 4-layer, 4-head Transformer to predict 5 normalized future waypoints with yaw (len_traj_pred: 5, learn_angle: True) and temporal distance. train/config/nomad.yaml uses a 256-D EfficientNet-B0 + 4-layer Transformer encoder (image_size: [96, 96], context_size: 3, goal_mask_prob: 0.5) feeding a 1D U-Net diffusion decoder (down_dims: [64, 128, 256], num_diffusion_iters: 10, len_traj_pred: 8) that depends on real-stanford/diffusion_policy. Exact parameter counts and checkpoint file sizes are not printed in the README or YAML configs. [1] [4]

Onboard Jetson deployment workflow and upstream-reported benchmarks

The repository ships a ROS Noetic deployment pipeline tested on a LoCoBot with an NVIDIA Jetson Orin Nano, where deployment/config/robot.yaml configures a frame_rate observation rate of 4 Hz (a configuration setting rather than a measured inference throughput benchmark) and pd_controller.py converts /waypoint outputs to /cmd_vel_mux/input/navi velocity commands alongside a joystick deadman switch. To deploy, the README directs users to activate the vint_deployment conda environment, place the chosen .pth checkpoint in deployment/model_weights/, record a demonstration ROS bag (record_bag.sh) and subsample it into a topological image map (create_topomap.sh), and then launch goal-conditioned navigation via navigate.sh; for goal-agnostic exploration without a target goal, only NoMaD is supported via exploration.sh / explore.sh. Upstream notes independent deployments on Clearpath Jackal, Unitree A1, DJI Tello, TurtleBot2, Vizbot, and CARLA. In arXiv:2310.07896v1 and arXiv:2306.14846v2, the authors report positive cross-embodiment transfer for ViNT over single-dataset specialists and improved navigation performance with lower collision rates for NoMaD against five baselines on real mobile robots. All reported results are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [3] [4] [5] [6]

Open MIT code, unversioned Drive weights, and partial public data

The repository code and documentation are released under the MIT License (LICENSE at dca79815b704). However, the four official PyTorch checkpoints (nomad.pth, vint.pth, vint_late_fusion.pth, and gnm_large.pth, mapped in deployment/config/models.yaml) are distributed via an unversioned Google Drive folder without a separate weights license declaration, so checkpoint license coverage remains unresolved in the supplied evidence (public download access alone does not establish reuse terms). Pretraining combines five public datasets (RECON, TartanDrive, SCAND, GoStanford2, SACSoN/HuRoN)—each requiring separate review of its upstream terms—with unreleased private datasets such as Seattle, preventing full from-scratch reproduction from the publicly released data alone (upstream directs readers to contact the respective authors for access to unreleased data). [1] [2] [3] [4]

What we checked

None. Prepared from public upstream sources without hands-on training, inference, or robot execution.

Updates

Upstream last updated
Sep 15, 2024 [1]
Latest release
Not established
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
robodhruv/visualnav-transformer@dca79815b704e5aa9c6bdc3082351f9e3b2848c2 · software repository

Separate license matrix

MIT-licensed training and ROS deployment codebase with four publicly downloadable Google Drive checkpoints, classified as partial because checkpoints lack a separate license declaration on Google Drive and pretraining includes unreleased private datasets.

LayerAvailability / license
Codeavailable · MIT
[1] [2]
Model weightsavailable · License not fully assessed
Four .pth checkpoints (nomad.pth, vint.pth, vint_late_fusion.pth, gnm_large.pth) are publicly downloadable via Google Drive without a separate weights license file. [1] [3]
Training datasetsavailable · License not fully assessed
Partial public availability: five datasets (RECON, TartanDrive, SCAND, GoStanford2, SACSoN/HuRoN) are public with separate upstream licenses, while additional training datasets (e.g., Seattle) remain unreleased. [1] [4]
Documentationavailable · MIT
[1] [2]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile drafted from robodhruv/visualnav-transformer@dca79815b704.