TraceVLA
Robotics modelMIT-licensed Vision-Language-Action checkpoints (7.54B OpenVLA-based and Phi-3-Vision-based) fine-tuned with visual trace prompting, overlaying Co-Tracker 3 historical point trajectories onto RGB observations to improve spatiotemporal manipulation awareness. [1] [2] [3] [4]
Setup & usage →CheckpointsModel hub / homepageGetting started
- For
- Robotics researchers studying visual prompting and temporal history representations in VLAs; Practitioners benchmarking OpenVLA variants on SimplerEnv Fractal and Bridge [1] [3]
- Embodiments
- Google Robot (SimplerEnv Fractal), WidowX (SimplerEnv Bridge), Open X-Embodiment 7-DoF manipulators [1] [3] [4]
- Source rights
- Named licenses are documented by part. Review licenses
Source researched · Not tested by us
Sources checked · Research details
Overview
TraceVLA (umd-huang-lab/tracevla, redirected from FrankZheng2022/tracevla; arXiv:2412.10345) is a research Vision-Language-Action project from the University of Maryland and Microsoft Research that adds spatial-temporal history to single-frame VLAs via visual trace prompting. Instead of modifying the action decoder, TraceVLA runs Co-Tracker 3 (facebook/cotracker3 scaled_offline.pth) on incoming camera frames to draw historical motion trajectories of the end effector and moving objects, then passes both the raw image and the trace-overlaid image (separated by a special separator token) into an autoregressive VLA predicting single-step 7-DoF end-effector deltas. [1] [3] [4]
Model card summary
- Architecture
Two autoregressive VLA variants—TraceVLA-7B (OpenVLA-7B Prismatic VLM backbone) and TraceVLA-Phi3V (Phi-3-Vision-128K-Instruct backbone with CLIP ViT-L/14-336)—prompted with paired raw and Co-Tracker 3 trace-overlaid RGB images separated by a special separator token.
- Base model
openvla/openvla-7b (for TraceVLA-7B) and OpenVLA-Phi3V / microsoft/Phi-3-vision-128k-instruct (for TraceVLA-Phi3V)
- Parameter count
7,541,239,360 (~7.54B BF16 params for TraceVLA-7B per HF safetensors metadata); ~4.2B class for TraceVLA-Phi3V (Phi-3-Vision 7.72 GB .bin; exact scalar not exposed in HF metadata)
- Input modalities
- rgblanguage
Single-camera RGB observation paired with a Co-Tracker 3 trace-overlaid version of the same frame, plus English task instruction.
- Output modalities
- ee-pose-deltagripper-command
Single-step OpenVLA-style discretized 7-DoF end-effector delta actions unnormalized via dataset_statistics.json.
- Action space
Autoregressively decoded single-step 7-DoF end-effector delta + gripper actions following OpenVLA's discrete action-token formulation and per-dataset unnormalization.
- Trained / supported embodiments
- Google Robot (SimplerEnv Fractal)
- WidowX (SimplerEnv Bridge)
- Open X-Embodiment 7-DoF manipulators
- Training datasets
- Open X-Embodiment robot mixture augmented with Co-Tracker visual traces (trace-annotated Bridge and Fractal datasets marked '[Coming soon]' and not yet released)
Checkpoints
| Checkpoint | Format | Size | Access | Pinned revision |
|---|---|---|---|---|
| furonghuang-lab/tracevla_7b | safetensors | 14.05 GB | open | 23ac599e7fd9 |
| furonghuang-lab/tracevla_phi3v | pytorch | 7.72 GB | open | b879e42622f0 |
Both checkpoints are ungated and MIT-licensed on Hugging Face; inference also requires downloading external facebook/cotracker3 scaled_offline.pth.
Is this for you?
- Use cases
- Zero-shot manipulation inference in SimplerEnv (Google Robot Fractal and WidowX Bridge) using visual trace prompting
- Fine-tuning OpenVLA-7B or OpenVLA-Phi3V with historical point-trajectory overlays
- Comparing 7B Llama-2 vs compact Phi-3-Vision VLA backbones under identical trace prompting
- Known limitations
- Visual-trace annotated Bridge and Fractal training datasets are marked '[Coming soon]' in the README and have not been released
- Inference requires running an external Co-Tracker 3 checkpoint (facebook/cotracker3 scaled_offline.pth) online to overlay historical traces, falling back to a duplicate raw image when no valid trace is returned
- Documented Transformers inference examples configure trust_remote_code=True and flash_attention_2, and the Phi-3-Vision codebase lives on a separate phi3 git branch
- Repository has had no commits since 2025-01-08, and README clone URL references a non-existent openvla/tracevla path instead of umd-huang-lab/tracevla
- Safety and limitations
- Checkpoints require trust_remote_code=True in Hugging Face Transformers and depend on an external Co-Tracker 3 checkpoint (separate upstream license/terms).
- Visual-trace annotated training datasets have not been published, limiting independent pretraining reproduction.
- Research prototype without robot-side safety guardrails or active maintenance since January 2025.
Compute & fine-tuning
- Inference compute
- Documented hardware: CUDA GPU with FlashAttention 2 (exact model and VRAM not specified upstream)
- Precision:
bf16 - Edge / consumer capable: No
14.05 GB (7B) or 7.72 GB (Phi3V) BF16 weights plus Co-Tracker 3 point tracker; exact VRAM and latency_ms not reported in README.
- Fine-tuning
- Supported: Yes
- Methods: Multi-node torchrun full fine-tuning via vla-scripts/train.py (64gpu-tracevla preset)
- Recipe: Upstream fine-tuning documentation
Documented reproduction command uses 8 nodes x 8 GPUs; single-GPU LoRA recipe is not documented.
- Host computer
Single CUDA GPU for bfloat16 inference (14.05 GB safetensors for TraceVLA-7B, 7.72 GB PyTorch .bin for TraceVLA-Phi3V, plus Co-Tracker 3 overhead; exact minimum VRAM not stated); training script defaults to 8 nodes x 8 GPUs (64 GPUs).
Exact minimum GPU VRAM and inference latency are not documented in README or model cards.
- Operating system
- Linux
Standard Linux conda, CUDA, and flash-attn==2.5.5 workflow documented.
[1] · Editorial assessment
- Dependencies
- Python 3.10
- PyTorch / CUDA with flash-attn==2.5.5
- Hugging Face Transformers (with trust_remote_code=True)
- facebookresearch/co-tracker + facebook/cotracker3 scaled_offline.pth checkpoint
- Skills
- Python 3.10 and conda
- PyTorch, FlashAttention 2, and Hugging Face Transformers
- Co-Tracker 3 point-tracking setup and SimplerEnv evaluation
Inferred from documented Co-Tracker, flash-attn, and SimplerEnv setup.
[1] · Editorial assessment
Setup & usage
Open the upstream getting-started guide [1] · Model card / documentation [1]
- Create a Python 3.10 conda environment
- Clone facebookresearch/co-tracker, install it with pip install -e ., and download facebook/cotracker3 scaled_offline.pth
- Clone umd-huang-lab/tracevla (checkout phi3 branch for TraceVLA-Phi3V), run pip install -e ., and install flash-attn==2.5.5
- Load furonghuang-lab/tracevla_7b (or tracevla_phi3v) with AutoModelForCausalLM (trust_remote_code=True, bfloat16) and initialize TraceProcessor(cotracker_model_path)
Upstream-reported benchmarks
Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.
| Benchmark | Upstream-reported score | Scope & caveats |
|---|---|---|
| SimplerEnv Fractal (Google Robot, Visual Matching & Variant Aggregation) | TraceVLA-7B: 45.8% VM / 49.8% VA (vs OpenVLA-7B 36.5% VM / 35.8% VA); TraceVLA-Phi3V: 44.0% VM / 45.1% VA (vs OpenVLA-Phi3V 41.0% VM / 35.8% VA) | Upstream-reported averages across 4 Fractal tasks (Pick up Coke, Move near, Open/Close Drawer, Put in Drawer); TraceVLA-Phi3V scores 0% on Put in Drawer; not independently reproduced by OSR. |
| SimplerEnv Bridge (WidowX, 4 tasks) | TraceVLA-7B: 27.7% average (vs OpenVLA-7B 16.7%); TraceVLA-Phi3V: 21.9% average (vs OpenVLA-Phi3V 5.2%) | Upstream-reported averages across Put Spoon, Put Carrot, Stack Block, Put Eggplant; TraceVLA-Phi3V scores 0% on Put Carrot; not independently reproduced by OSR. |
Project notes
Two MIT checkpoints and Co-Tracker runtime coupling
Two ungated MIT-licensed checkpoints are published on Hugging Face: furonghuang-lab/tracevla_7b (7,541,239,360 BF16 parameters, 14.05 GB safetensors, fine-tuned from openvla/openvla-7b) and furonghuang-lab/tracevla_phi3v (7.72 GB PyTorch .bin, Phi3VForCausalLM with CLIP ViT-L/14-336, backed by the repository's separate phi3 branch). Both documented Transformers inference examples configure trust_remote_code=True and _attn_implementation='flash_attention_2', alongside an online TraceProcessor backed by Co-Tracker 3. In the TraceVLA-7B inference example, when Co-Tracker does not return a valid trace, the code falls back to duplicating the raw image ([image, image]) with the standard OpenVLA prompt; in the TraceVLA-Phi3V example, images are resized to 256x256 for Co-Tracker and 336x336 for the Phi-3-Vision encoder, and both the no-trace and trace branches use Phi3V chat-template prompts while passing [image, image_overlaid]. [1] [3] [4]
Upstream SimplerEnv results and documentation gaps
In the repository README and Hugging Face model cards, upstream reports SimplerEnv gains over base OpenVLA checkpoints: TraceVLA-7B reaches 45.8% Visual Matching and 49.8% Variant Aggregation on Fractal (versus 36.5% / 35.8% for OpenVLA-7B) and 27.7% average on Bridge (versus 16.7%), while TraceVLA-Phi3V reaches 44.0% VM / 45.1% VA on Fractal and 21.9% on Bridge (noting 0% on Fractal Put in Drawer and Bridge Put Carrot). Neither the README nor the model cards specify minimum inference VRAM, Co-Tracker latency overhead, or control frequency (Hz), and the documented training command assumes an 8-node × 8-GPU (64-GPU) cluster without a single-GPU LoRA recipe. All benchmark figures are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [3] [4]
What we checked
None. Prepared from public upstream sources and Hugging Face API metadata without hands-on training, fine-tuning, or inference.
Not yet established
- Documented closed-loop control frequency (Hz)
Updates
- Upstream last updated
- Jan 8, 2025 [1]
- Profile updated
- Nov 21, 2026
- Sources last checked
- Nov 21, 2026
- Next review
- Feb 19, 2027
- Reviewed revision
- umd-huang-lab/tracevla@d5454af9ebe51e996005e13061ac3f6964e2923f; HF furonghuang-lab/tracevla_7b@23ac599e7fd95b1fb26dc9b93d03be4ad60d136f · combined hardware and software repository
Separate license matrix
MIT-licensed visual-trace-prompted VLA codebase and two ungated MIT checkpoints (7B OpenVLA and Phi-3-Vision); classified as partial openness because the visual-trace annotated training datasets remain unreleased and runtime requires external Co-Tracker 3 weights.
| Layer | Availability / license |
|---|---|
| Code | available · MIT Verified MIT License in root LICENSE file. [1] [2] |
| Model weights | available · MIT Both furonghuang-lab/tracevla_7b and furonghuang-lab/tracevla_phi3v declare license: mit in HF YAML frontmatter; runtime also requires external facebook/cotracker3 weights. [3] [4] |
| Training datasets | unavailable · Upstream Open X-Embodiment is public (mixed licenses), but TraceVLA's visual-trace annotated Bridge and Fractal datasets remain unreleased ('[Coming soon]') TraceVLA-specific visual-trace annotated datasets are marked '[Coming soon]' and not yet available. [1] |
| Documentation | available · MIT [1] [2] |
Maintenance status, licensing, and unreleased trace data
Both the umd-huang-lab/tracevla codebase (verified in the root LICENSE file) and the two Hugging Face checkpoints declare the MIT License. However, the repository has had no commits since January 8, 2025, its README clone snippet still points to a non-existent openvla/tracevla URL, and the visual-trace annotated Bridge and Fractal training datasets remain marked [Coming soon]. Teams evaluating TraceVLA should also review the separate upstream terms for facebook/cotracker3 and the underlying openvla/openvla-7b (Llama-2) or Phi-3-vision base models. [1] [2] [3] [4]
References
Each source links to the exact revision we read. “Used for” shows where this page relies on it.
- S001 umd-huang-lab/tracevla GitHub repository metadata and README.md
Documents Co-Tracker 3 dependency, TraceVLA-7B inference snippet, 64-GPU training script, SimplerEnv Fractal/Bridge results, and '[Coming soon]' status for trace-annotated datasets.Revision, date & where cited
UMD Huang Lab (GitHub) · github.com · revision
d5454af· GitHub API & README.md at commit d5454af9ebe5 (2025-01-08); FrankZheng2022/tracevla redirects here · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Is this for you?, Compute & fine-tuning and 6 more - S002 umd-huang-lab/tracevla root LICENSE at pinned commit
MIT License (Copyright 2024 FrankZheng2022).Revision, date & where cited
FrankZheng2022 · github.com · revision
d5454af· LICENSE at commit d5454af9ebe5 · Accessed Nov 21, 2026
Used for: Separate license matrix - S003 Hugging Face model card and API metadata: furonghuang-lab/tracevla_7b
7,541,239,360 BF16 parameters, 14.05 GB ungated safetensors, license: mit, fine-tuned from openvla/openvla-7b; reports SimplerEnv Fractal 45.8% VM / 49.8% VA and Bridge 27.7%.Revision, date & where cited
furonghuang-lab (Hugging Face) · huggingface.co · HF API & README.md at revision 23ac599e7fd9 (lastModified 2025-01-07) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 7 more - S004 Hugging Face model card, config.json, and API metadata: furonghuang-lab/tracevla_phi3v
7.72 GB ungated PyTorch .bin checkpoint, license: mit, Phi3VForCausalLM (CLIP ViT-L/14-336 + 32-layer hidden_size 3072); reports SimplerEnv Fractal 44.0% VM / 45.1% VA and Bridge 21.9%.Revision, date & where cited
furonghuang-lab (Hugging Face) · huggingface.co · HF API, README.md, and config.json at revision b879e42622f0 (lastModified 2025-01-07) · Accessed Nov 21, 2026
Used for: Overview, Model card summary, Checkpoints, Is this for you? and 7 more
Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1
Change history
2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from umd-huang-lab/tracevla@d5454af9ebe5 and HF tracevla_7b + tracevla_phi3v checkpoints.