VISTA visual-tactile policy
Robotics modelEquivariant diffusion policy combining wrist vision and tactile observations for contact-rich manipulation. [1]
Setup & usage →CheckpointsModel hub / homepageSource codeGetting started
- For
- Manipulation researchers with tactile sensors or matching simulation infrastructure. [1]
- Parameters
- Not stated in upstream README or arXiv:2610.03333 abstract (trained per task from scratch [1] [6] [7]
- Embodiments
- 8 Isaac Sim / Isaac Lab / TacEx simulated contact-rich tasks (grasp_classify, insert_HDMI, insert_hole, insert_tube,… [1]
- Source rights
- Named licenses are documented by part. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Licenses & related hardware
- Ecosystem
- Isaac Sim
Overview
VISTA (Kenn3o3/Vista, CoRL 2026, arXiv:2610.03333) is a workspace-level finite-group equivariant visual-tactile diffusion policy (SO(3) I60 and SO(2) C8 variants) for contact-rich manipulation. It lifts wrist RGB and tactile observations onto spherical tokens, fuses them via permutation-equivariant spherical fusion, aligns the harmonic representation using end-effector orientation, and decodes action chunks with DDIM diffusion. [1] [6]
Model card summary
- Architecture
Workspace-level finite-group equivariant visual-tactile diffusion policy (SO(3) I60 and SO(2) C8 variants) that lifts wrist RGB and tactile images onto spherical tokens, fuses them via permutation-equivariant spherical fusion, rotates harmonics by end-effector orientation, and decodes action chunks via DDIM diffusion.
- Parameter count
Not stated in upstream README or arXiv:2610.03333 abstract (trained per task from scratch; no pretrained VISTA policy checkpoints released)
No pretrained VISTA policy weights are published to count parameters, and neither the README nor the arXiv abstract reports a parameter count.
- Input modalities
- rgbtactileproprioception
Wrist RGB camera, tactile sensor images/features, and end-effector orientation.
- Output modalities
- ee-pose-deltagripper-command
Predicts 8-step end-effector manipulation action chunks (horizon: 16, n_action_steps: 8, 8 DDIM steps in vista_exec8_ddim8) for contact-rich tasks.
- Action space
Continuous finite-group equivariant diffusion action chunks (horizon: 16, n_obs_steps: 2, n_action_steps: 8; main profile vista_exec8_ddim8 executes 8 actions per replan using 8 DDIM steps) for contact-rich end-effector manipulation.
- Trained / supported embodiments
- 8 Isaac Sim / Isaac Lab / TacEx simulated contact-rich tasks (grasp_classify, insert_HDMI, insert_hole, insert_tube, lift_bottle, lift_can, pull_out_key, put_bottle_in_shelf)
- Franka real-robot setup (separate Kenn3o3/vt_franka repository linked by upstream; not verified here)
- Training datasets
- KLHWONG/MIDFOV_EXPERIMENT (Hugging Face / ModelScope, Apache-2.0, HDF5 demonstrations across 8 simulated contact-rich tasks)
- Interfaces
- Shell/Python CLI (scripts/train_vista.sh, univtac.eval.runner, offline_inference_smoke)
- HDF5 episodes
- PyTorch .ckpt checkpoints
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| VISTA per-task policy checkpoints (not publicly released; trained locally via scripts/train_vista.sh) | pytorch | Not stated | weights-not-released | Not pinned |
Upstream provides training and offline-inference scripts (writing best.ckpt locally) and baseline ACT/ViTAL encoder weights on ModelScope, but does not release pretrained VISTA policy weights.
Is this for you?
- Use cases
- Contact-rich policy training
- Visual-tactile simulation evaluation
- Known limitations
- Inherited ISP code has academic/educational-use terms despite the root Apache license.
- Large simulation assets and compiled libraries are external.
- Separate real-robot repository not tested.
- Safety and limitations
- No pretrained VISTA policy checkpoints are publicly released; users must train checkpoints locally (and README notes checkpoints contain pickled Python objects—load only trusted files).
- Root license is Apache-2.0, but THIRD_PARTY_NOTICES.md documents academic-research-and-educational-use restrictions on inherited ISP code portions.
- Checked-in environment.yml is an older Python 3.9 lock and must not be used; simulation requires Isaac Sim 4.5, Isaac Lab 2.1.1, and a vcpkg/libuipc build.
- Separate real-robot repository (Kenn3o3/vt_franka) was not inspected or tested; Open Source Robotics has not run training, offline inference, or simulation rollouts.
Compute & fine-tuning
- Inference compute
- Documented hardware: Linux with NVIDIA GPU (CUDA 12.4 / PyTorch 2.5.1; Isaac Sim 4.5 + Isaac Lab 2.1.1 + libuipc for simulation rollouts)
- Edge / consumer capable: No
Exact minimum VRAM and inference latency_ms are not established in the README.
[1] · Editorial assessment
- Fine-tuning
- Supported: Yes
- Methods: Per-task training from scratch via scripts/train_vista.sh (so3 / so2 configs; paper setting is 50 demos, 300 epochs)
- Recipe: Upstream fine-tuning documentation
- Host computer
Linux with NVIDIA GPU for simulation; exact training VRAM not established.
- Operating system
- Linux
README states Linux with an NVIDIA GPU is required for simulation.
- Dependencies
- Isaac Sim 4.5
- Isaac Lab 2.1.1
- libuipc
- MIDFOV dataset
Setup & usage
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| MIDFOV_EXPERIMENT 8-task simulated contact-rich suite & real-robot evaluations (CoRL 2026, arXiv:2610.03333) | See arXiv:2610.03333 (paper abstract reports improved data efficiency over visuotactile imitation learning baselines in simulation and real-world robotic settings; per-task scores and baseline tables are in the paper) | Upstream-reported CoRL 2026 abstract summary; exact per-task tables and baseline comparisons were not extracted, no pretrained VISTA checkpoints are released, and OSR has not run training, offline inference, or simulation rollouts. |
Project notes
Action space, checkpoints, and environment setup
In configs/vista/so3.yaml at pinned commit a0381fb1f95498dda9b006344bbbe923d42b5ccc, VISTA uses horizon: 16, n_obs_steps: 2, and n_action_steps: 8; the primary evaluation profile vista_exec8_ddim8 executes 8 actions per replan using 8 DDIM steps. Minimum training/inference VRAM and inference latency are not established by the reviewed evidence, and no closed-loop control rate should be inferred from the eight-action execution profile. Crucially, upstream does not release pretrained VISTA policy checkpoints (verified across the README and Hugging Face author KLHWONG, which hosts only the dataset): users train per-task checkpoints locally (scripts/train_vista.sh, 50 demos / 300 epochs for the paper setting) from the linked KLHWONG/MIDFOV_EXPERIMENT HDF5 dataset, while baseline ACT/ViTAL encoder weights are hosted separately on ModelScope. Upstream also warns that checkpoints contain pickled Python objects and should only be loaded from trusted sources. [1] [5] [7]
The README requires Linux with an NVIDIA GPU, Python 3.10, PyTorch 2.5.1 (CUDA 12.4), Isaac Sim 4.5, Isaac Lab 2.1.1, and a bootstrapped vcpkg libuipc build, explicitly warning not to install the older Python 3.9 environment.yml. Training and the documented offline-inference finiteness check run without starting Isaac Sim; headless rollouts cover eight simulated contact-rich tasks, with real-robot Franka code linked in a separate Kenn3o3/vt_franka repository. [1] [4]
Upstream-reported benchmarks and unverified runtime
In the arXiv:2610.03333 abstract, the authors report experiments in both simulation and real-world robotic settings showing improved data efficiency over visuotactile imitation learning baselines (the repository separately provides training/evaluation wrappers for ACT, ISP, ManiFeel, EquiDiff+Tac, and ViTAL baselines, though exact per-task tables and baseline comparisons were not extracted here). All evaluation claims are strictly upstream-reported; Open Source Robotics has not installed dependencies, downloaded datasets, trained checkpoints, or run offline inference or simulation rollouts. [1] [6]
What we checked
Source research only. No commercial relationship or hands-on testing. No upstream media copied to the public site.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Upstream last updated
- Oct 5, 2026 [7]
- Latest release
- Not established
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- GitHub Kenn3o3/Vista@a0381fb1f95498dda9b006344bbbe923d42b5ccc; HF dataset KLHWONG/MIDFOV_EXPERIMENT@691dc1df1e2a5fffee62e3568417eebc41f535ac · combined hardware and software repository
Separate license matrix
Inspectable contact-rich robot policy and task dataset; inherited implementation restrictions and external assets prevent a blanket open-source claim.
| Layer | Availability / license |
|---|---|
| Code | available · Apache-2.0 root; inherited ISP portions academic/educational only Artifact and declaration inspected at the cited scope. External assets/dependencies and actual operation need separate verification. [1] [3] |
| Model weights | unavailable · License not fully assessed Upstream repository supplies training and offline-inference code but does not release pretrained VISTA policy checkpoints (no model repos under HF author KLHWONG); baseline ACT/ViTAL encoder weights are hosted separately on ModelScope. [1] [7] |
| Training datasets | available · Apache-2.0 dataset-card declaration Artifact and declaration inspected at the cited scope. External assets/dependencies and actual operation need separate verification. [5] |
| Documentation | available · Apache-2.0 root with third-party notices Artifact and declaration inspected at the cited scope. External assets/dependencies and actual operation need separate verification. [1] [3] |
Mixed rights across code, inherited modules, and dataset
While the repository root carries an Apache-2.0 license and the KLHWONG/MIDFOV_EXPERIMENT Hugging Face dataset card declares Apache-2.0, THIRD_PARTY_NOTICES.md specifies that inherited ISP implementation portions retain academic-research-and-educational-use restrictions. Large simulator assets and compiled libraries are external to the GitHub repository, so VISTA should be treated as mixed-rights source-available material rather than a uniformly permissive commercial package. [1] [2] [3] [5]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 VISTA setup and workflow
Revision, date & where cited
Kenn3o3/Vista contributors · github.com · revision
a0381fb· README.md at · Accessed Nov 7, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 6 more - S002 Root Apache-2.0 license
Revision, date & where cited
Kenn3o3/Vista contributors · github.com · revision
a0381fb· LICENSE at · Accessed Nov 7, 2026
Used for: Separate license matrix - S003 Inherited code restrictions
Revision, date & where cited
Kenn3o3/Vista contributors · github.com · revision
a0381fb· THIRD_PARTY_NOTICES.md at · Accessed Nov 7, 2026
Used for: Is this for you?, Separate license matrix - S004 Simulation notes
Revision, date & where cited
Kenn3o3/Vista contributors · github.com · revision
a0381fb· docs/SIMULATION.md at · Accessed Nov 7, 2026
Used for: Is this for you?, Setup & usage, Action space, checkpoints, and environment setup - S005 MIDFOV dataset card
Revision, date & where cited
VISTA authors · huggingface.co · Dataset card at · Accessed Nov 7, 2026
Used for: Model card summary, Is this for you?, Setup & usage, Action space, checkpoints, and environment setup and 1 more - S006 Equivariant Visual-Tactile Diffusion Policy for Contact-Rich Manipulation (arXiv:2610.03333)
Workspace-level equivariant visuotactile diffusion policy projecting visual and tactile observations into spherical tokens, fusing via permutation-equivariant spherical fusion, rotating harmonics by end-effector orientation, and decoding actions with an equivariant diffusion policy.Revision, date & where cited
VISTA authors (CoRL 2026 / arXiv) · arxiv.org · arXiv:2610.03333v1 abstract · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Upstream-reported benchmarks, Upstream-reported benchmarks and unverified runtime - S007 VISTA so3.yaml config and GitHub/Hugging Face API metadata
so3.yaml sets horizon: 16, n_obs_steps: 2, n_action_steps: 8, action_horizon: 8; GitHub pushed_at 2026-10-05T16:05:03Z, SPDX Apache-2.0; HF author KLHWONG has no model repos (dataset KLHWONG/MIDFOV_EXPERIMENT only).Revision, date & where cited
Kenn3o3/Vista contributors (GitHub / Hugging Face) · github.com · revision
a0381fb· configs/vista/so3.yaml at ; GitHub API; HF API · Accessed Nov 21, 2026
Used for: Model card summary, Checkpoints, Is this for you?, Action space, checkpoints, and environment setup and 2 more
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Migrated from 0.7.0-draft.5 project to 0.8.0-draft.1 model; added equivariant spherical diffusion architecture, weights-not-released checkpoint record, and upstream-reported CoRL 2026 benchmarks.