Open Source RoboticsRobots, software & learning

VISTA visual-tactile policy

Robotics model

Equivariant diffusion policy combining wrist vision and tactile observations for contact-rich manipulation. [1]

Setup & usage →CheckpointsModel hub / homepageSource codeGetting started

For
Manipulation researchers with tactile sensors or matching simulation infrastructure. [1]
Parameters
Not stated in upstream README or arXiv:2610.03333 abstract (trained per task from scratch [1] [6] [7]
Embodiments
8 Isaac Sim / Isaac Lab / TacEx simulated contact-rich tasks (grasp_classify, insert_HDMI, insert_hole, insert_tube,… [1]
Source rights
Named licenses are documented by part. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware
Ecosystem
Isaac Sim

Requirements · Research details and dates

On this page

Overview

VISTA (Kenn3o3/Vista, CoRL 2026, arXiv:2610.03333) is a workspace-level finite-group equivariant visual-tactile diffusion policy (SO(3) I60 and SO(2) C8 variants) for contact-rich manipulation. It lifts wrist RGB and tactile observations onto spherical tokens, fuses them via permutation-equivariant spherical fusion, aligns the harmonic representation using end-effector orientation, and decodes action chunks with DDIM diffusion. [1] [6]

Model card summary

Architecture

Workspace-level finite-group equivariant visual-tactile diffusion policy (SO(3) I60 and SO(2) C8 variants) that lifts wrist RGB and tactile images onto spherical tokens, fuses them via permutation-equivariant spherical fusion, rotates harmonics by end-effector orientation, and decodes action chunks via DDIM diffusion.

[1] [6]

Parameter count

Not stated in upstream README or arXiv:2610.03333 abstract (trained per task from scratch; no pretrained VISTA policy checkpoints released)

No pretrained VISTA policy weights are published to count parameters, and neither the README nor the arXiv abstract reports a parameter count.

[1] [6] [7] · Editorial assessment

Input modalities
rgbtactileproprioception

Wrist RGB camera, tactile sensor images/features, and end-effector orientation.

[1] [6]

Output modalities
ee-pose-deltagripper-command

Predicts 8-step end-effector manipulation action chunks (horizon: 16, n_action_steps: 8, 8 DDIM steps in vista_exec8_ddim8) for contact-rich tasks.

[1] [6] [7] · Editorial assessment

Action space

Continuous finite-group equivariant diffusion action chunks (horizon: 16, n_obs_steps: 2, n_action_steps: 8; main profile vista_exec8_ddim8 executes 8 actions per replan using 8 DDIM steps) for contact-rich end-effector manipulation.

[1] [7]

Trained / supported embodiments
  • 8 Isaac Sim / Isaac Lab / TacEx simulated contact-rich tasks (grasp_classify, insert_HDMI, insert_hole, insert_tube, lift_bottle, lift_can, pull_out_key, put_bottle_in_shelf)
  • Franka real-robot setup (separate Kenn3o3/vt_franka repository linked by upstream; not verified here)

[1]

Training datasets
  • KLHWONG/MIDFOV_EXPERIMENT (Hugging Face / ModelScope, Apache-2.0, HDF5 demonstrations across 8 simulated contact-rich tasks)

[1] [5]

Interfaces
  • Shell/Python CLI (scripts/train_vista.sh, univtac.eval.runner, offline_inference_smoke)
  • HDF5 episodes
  • PyTorch .ckpt checkpoints

[1]

Checkpoints

CheckpointFormatSizeAccessPinned revision
VISTA per-task policy checkpoints (not publicly released; trained locally via scripts/train_vista.sh)pytorchNot statedweights-not-releasedNot pinned

Upstream provides training and offline-inference scripts (writing best.ckpt locally) and baseline ACT/ViTAL encoder weights on ModelScope, but does not release pretrained VISTA policy weights.

[1] [7]

Is this for you?

Use cases
  • Contact-rich policy training
  • Visual-tactile simulation evaluation

[1]

Known limitations
  • Inherited ISP code has academic/educational-use terms despite the root Apache license.
  • Large simulation assets and compiled libraries are external.
  • Separate real-robot repository not tested.

[1] [3] [5]

Safety and limitations
  • No pretrained VISTA policy checkpoints are publicly released; users must train checkpoints locally (and README notes checkpoints contain pickled Python objects—load only trusted files).
  • Root license is Apache-2.0, but THIRD_PARTY_NOTICES.md documents academic-research-and-educational-use restrictions on inherited ISP code portions.
  • Checked-in environment.yml is an older Python 3.9 lock and must not be used; simulation requires Isaac Sim 4.5, Isaac Lab 2.1.1, and a vcpkg/libuipc build.
  • Separate real-robot repository (Kenn3o3/vt_franka) was not inspected or tested; Open Source Robotics has not run training, offline inference, or simulation rollouts.

[1] [3] [4] [7]

Compute & fine-tuning

Inference compute
  • Documented hardware: Linux with NVIDIA GPU (CUDA 12.4 / PyTorch 2.5.1; Isaac Sim 4.5 + Isaac Lab 2.1.1 + libuipc for simulation rollouts)
  • Edge / consumer capable: No

Exact minimum VRAM and inference latency_ms are not established in the README.

[1] · Editorial assessment

Fine-tuning
  • Supported: Yes
  • Methods: Per-task training from scratch via scripts/train_vista.sh (so3 / so2 configs; paper setting is 50 demos, 300 epochs)
  • Recipe: Upstream fine-tuning documentation

[1]

Host computer

Linux with NVIDIA GPU for simulation; exact training VRAM not established.

[1]

Operating system
  • Linux

README states Linux with an NVIDIA GPU is required for simulation.

[1]

Dependencies
  • Isaac Sim 4.5
  • Isaac Lab 2.1.1
  • libuipc
  • MIDFOV dataset

[1]

Setup & usage

Open the upstream getting-started guide [1]

  • Use the README's Python 3.10 environment, not the older environment.yml lock.
  • Download the task HDF5 data and required simulator assets.
  • Train a checkpoint and attempt the documented offline-inference check before simulation rollout.

[1] [4] [5]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
MIDFOV_EXPERIMENT 8-task simulated contact-rich suite & real-robot evaluations (CoRL 2026, arXiv:2610.03333)See arXiv:2610.03333 (paper abstract reports improved data efficiency over visuotactile imitation learning baselines in simulation and real-world robotic settings; per-task scores and baseline tables are in the paper)Upstream-reported CoRL 2026 abstract summary; exact per-task tables and baseline comparisons were not extracted, no pretrained VISTA checkpoints are released, and OSR has not run training, offline inference, or simulation rollouts.

[1] [6]

Project notes

Action space, checkpoints, and environment setup

In configs/vista/so3.yaml at pinned commit a0381fb1f95498dda9b006344bbbe923d42b5ccc, VISTA uses horizon: 16, n_obs_steps: 2, and n_action_steps: 8; the primary evaluation profile vista_exec8_ddim8 executes 8 actions per replan using 8 DDIM steps. Minimum training/inference VRAM and inference latency are not established by the reviewed evidence, and no closed-loop control rate should be inferred from the eight-action execution profile. Crucially, upstream does not release pretrained VISTA policy checkpoints (verified across the README and Hugging Face author KLHWONG, which hosts only the dataset): users train per-task checkpoints locally (scripts/train_vista.sh, 50 demos / 300 epochs for the paper setting) from the linked KLHWONG/MIDFOV_EXPERIMENT HDF5 dataset, while baseline ACT/ViTAL encoder weights are hosted separately on ModelScope. Upstream also warns that checkpoints contain pickled Python objects and should only be loaded from trusted sources. [1] [5] [7]

The README requires Linux with an NVIDIA GPU, Python 3.10, PyTorch 2.5.1 (CUDA 12.4), Isaac Sim 4.5, Isaac Lab 2.1.1, and a bootstrapped vcpkg libuipc build, explicitly warning not to install the older Python 3.9 environment.yml. Training and the documented offline-inference finiteness check run without starting Isaac Sim; headless rollouts cover eight simulated contact-rich tasks, with real-robot Franka code linked in a separate Kenn3o3/vt_franka repository. [1] [4]

Upstream-reported benchmarks and unverified runtime

In the arXiv:2610.03333 abstract, the authors report experiments in both simulation and real-world robotic settings showing improved data efficiency over visuotactile imitation learning baselines (the repository separately provides training/evaluation wrappers for ACT, ISP, ManiFeel, EquiDiff+Tac, and ViTAL baselines, though exact per-task tables and baseline comparisons were not extracted here). All evaluation claims are strictly upstream-reported; Open Source Robotics has not installed dependencies, downloaded datasets, trained checkpoints, or run offline inference or simulation rollouts. [1] [6]

What we checked

Source research only. No commercial relationship or hands-on testing. No upstream media copied to the public site.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Oct 5, 2026 [7]
Latest release
Not established
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
GitHub Kenn3o3/Vista@a0381fb1f95498dda9b006344bbbe923d42b5ccc; HF dataset KLHWONG/MIDFOV_EXPERIMENT@691dc1df1e2a5fffee62e3568417eebc41f535ac · combined hardware and software repository

Separate license matrix

Inspectable contact-rich robot policy and task dataset; inherited implementation restrictions and external assets prevent a blanket open-source claim.

LayerAvailability / license
Codeavailable · Apache-2.0 root; inherited ISP portions academic/educational only
Artifact and declaration inspected at the cited scope. External assets/dependencies and actual operation need separate verification. [1] [3]
Model weightsunavailable · License not fully assessed
Upstream repository supplies training and offline-inference code but does not release pretrained VISTA policy checkpoints (no model repos under HF author KLHWONG); baseline ACT/ViTAL encoder weights are hosted separately on ModelScope. [1] [7]
Training datasetsavailable · Apache-2.0 dataset-card declaration
Artifact and declaration inspected at the cited scope. External assets/dependencies and actual operation need separate verification. [5]
Documentationavailable · Apache-2.0 root with third-party notices
Artifact and declaration inspected at the cited scope. External assets/dependencies and actual operation need separate verification. [1] [3]

Mixed rights across code, inherited modules, and dataset

While the repository root carries an Apache-2.0 license and the KLHWONG/MIDFOV_EXPERIMENT Hugging Face dataset card declares Apache-2.0, THIRD_PARTY_NOTICES.md specifies that inherited ISP implementation portions retain academic-research-and-educational-use restrictions. Large simulator assets and compiled libraries are external to the GitHub repository, so VISTA should be treated as mixed-rights source-available material rather than a uniformly permissive commercial package. [1] [2] [3] [5]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Migrated from 0.7.0-draft.5 project to 0.8.0-draft.1 model; added equivariant spherical diffusion architecture, weights-not-released checkpoint record, and upstream-reported CoRL 2026 benchmarks.