Open Source RoboticsRobots, software & learning

TraceVLA

Robotics model

MIT-licensed Vision-Language-Action checkpoints (7.54B OpenVLA-based and Phi-3-Vision-based) fine-tuned with visual trace prompting, overlaying Co-Tracker 3 historical point trajectories onto RGB observations to improve spatiotemporal manipulation awareness. [1] [2] [3] [4]

Setup & usage →CheckpointsModel hub / homepageGetting started

For
Robotics researchers studying visual prompting and temporal history representations in VLAs; Practitioners benchmarking OpenVLA variants on SimplerEnv Fractal and Bridge [1] [3]
Parameters
7,541,239,360 (~7.54B BF16 params for TraceVLA-7B per HF safetensors metadata) [3] [4]
Embodiments
Google Robot (SimplerEnv Fractal), WidowX (SimplerEnv Bridge), Open X-Embodiment 7-DoF manipulators [1] [3] [4]
Source rights
Named licenses are documented by part. Review licenses

Source researched · Not tested by us
Sources checked · Research details

Licenses & related hardware

Requirements · Research details and dates

On this page

Overview

TraceVLA (umd-huang-lab/tracevla, redirected from FrankZheng2022/tracevla; arXiv:2412.10345) is a research Vision-Language-Action project from the University of Maryland and Microsoft Research that adds spatial-temporal history to single-frame VLAs via visual trace prompting. Instead of modifying the action decoder, TraceVLA runs Co-Tracker 3 (facebook/cotracker3 scaled_offline.pth) on incoming camera frames to draw historical motion trajectories of the end effector and moving objects, then passes both the raw image and the trace-overlaid image (separated by a special separator token) into an autoregressive VLA predicting single-step 7-DoF end-effector deltas. [1] [3] [4]

Model card summary

Architecture

Two autoregressive VLA variants—TraceVLA-7B (OpenVLA-7B Prismatic VLM backbone) and TraceVLA-Phi3V (Phi-3-Vision-128K-Instruct backbone with CLIP ViT-L/14-336)—prompted with paired raw and Co-Tracker 3 trace-overlaid RGB images separated by a special separator token.

[1] [3] [4]

Base model

openvla/openvla-7b (for TraceVLA-7B) and OpenVLA-Phi3V / microsoft/Phi-3-vision-128k-instruct (for TraceVLA-Phi3V)

[3] [4]

Parameter count

7,541,239,360 (~7.54B BF16 params for TraceVLA-7B per HF safetensors metadata); ~4.2B class for TraceVLA-Phi3V (Phi-3-Vision 7.72 GB .bin; exact scalar not exposed in HF metadata)

[3] [4]

Input modalities
rgblanguage

Single-camera RGB observation paired with a Co-Tracker 3 trace-overlaid version of the same frame, plus English task instruction.

[1] [3] [4]

Output modalities
ee-pose-deltagripper-command

Single-step OpenVLA-style discretized 7-DoF end-effector delta actions unnormalized via dataset_statistics.json.

[1] [3]

Action space

Autoregressively decoded single-step 7-DoF end-effector delta + gripper actions following OpenVLA's discrete action-token formulation and per-dataset unnormalization.

[1] [3] [4]

Trained / supported embodiments
  • Google Robot (SimplerEnv Fractal)
  • WidowX (SimplerEnv Bridge)
  • Open X-Embodiment 7-DoF manipulators

[1] [3] [4]

Training datasets
  • Open X-Embodiment robot mixture augmented with Co-Tracker visual traces (trace-annotated Bridge and Fractal datasets marked '[Coming soon]' and not yet released)

[1] [4]

Interfaces
  • Hugging Face Transformers AutoModelForCausalLM / AutoProcessor (predict_action)
  • TraceProcessor visual trace overlay wrapper (prismatic/eval/trace_processor.py)
  • SimplerEnv policy wrapper (prismatic/eval)
  • torchrun training CLI (vla-scripts/train.py)

[1] [3]

Checkpoints

CheckpointFormatSizeAccessPinned revision
furonghuang-lab/tracevla_7bsafetensors14.05 GBopen23ac599e7fd9
furonghuang-lab/tracevla_phi3vpytorch7.72 GBopenb879e42622f0

Both checkpoints are ungated and MIT-licensed on Hugging Face; inference also requires downloading external facebook/cotracker3 scaled_offline.pth.

[3] [4]

Is this for you?

Use cases
  • Zero-shot manipulation inference in SimplerEnv (Google Robot Fractal and WidowX Bridge) using visual trace prompting
  • Fine-tuning OpenVLA-7B or OpenVLA-Phi3V with historical point-trajectory overlays
  • Comparing 7B Llama-2 vs compact Phi-3-Vision VLA backbones under identical trace prompting

[1] [3] [4]

Known limitations
  • Visual-trace annotated Bridge and Fractal training datasets are marked '[Coming soon]' in the README and have not been released
  • Inference requires running an external Co-Tracker 3 checkpoint (facebook/cotracker3 scaled_offline.pth) online to overlay historical traces, falling back to a duplicate raw image when no valid trace is returned
  • Documented Transformers inference examples configure trust_remote_code=True and flash_attention_2, and the Phi-3-Vision codebase lives on a separate phi3 git branch
  • Repository has had no commits since 2025-01-08, and README clone URL references a non-existent openvla/tracevla path instead of umd-huang-lab/tracevla

[1] [3] [4]

Safety and limitations
  • Checkpoints require trust_remote_code=True in Hugging Face Transformers and depend on an external Co-Tracker 3 checkpoint (separate upstream license/terms).
  • Visual-trace annotated training datasets have not been published, limiting independent pretraining reproduction.
  • Research prototype without robot-side safety guardrails or active maintenance since January 2025.

[1] [3] [4]

Compute & fine-tuning

Inference compute
  • Documented hardware: CUDA GPU with FlashAttention 2 (exact model and VRAM not specified upstream)
  • Precision: bf16
  • Edge / consumer capable: No

14.05 GB (7B) or 7.72 GB (Phi3V) BF16 weights plus Co-Tracker 3 point tracker; exact VRAM and latency_ms not reported in README.

[1] [3] [4] · Editorial assessment

Fine-tuning

Documented reproduction command uses 8 nodes x 8 GPUs; single-GPU LoRA recipe is not documented.

[1]

Host computer

Single CUDA GPU for bfloat16 inference (14.05 GB safetensors for TraceVLA-7B, 7.72 GB PyTorch .bin for TraceVLA-Phi3V, plus Co-Tracker 3 overhead; exact minimum VRAM not stated); training script defaults to 8 nodes x 8 GPUs (64 GPUs).

Exact minimum GPU VRAM and inference latency are not documented in README or model cards.

[1] [3] [4] · Editorial assessment

Operating system
  • Linux

Standard Linux conda, CUDA, and flash-attn==2.5.5 workflow documented.

[1] · Editorial assessment

Dependencies
  • Python 3.10
  • PyTorch / CUDA with flash-attn==2.5.5
  • Hugging Face Transformers (with trust_remote_code=True)
  • facebookresearch/co-tracker + facebook/cotracker3 scaled_offline.pth checkpoint

[1] [3]

Skills
  • Python 3.10 and conda
  • PyTorch, FlashAttention 2, and Hugging Face Transformers
  • Co-Tracker 3 point-tracking setup and SimplerEnv evaluation

Inferred from documented Co-Tracker, flash-attn, and SimplerEnv setup.

[1] · Editorial assessment

Setup & usage

Open the upstream getting-started guide [1] · Model card / documentation [1]

  • Create a Python 3.10 conda environment
  • Clone facebookresearch/co-tracker, install it with pip install -e ., and download facebook/cotracker3 scaled_offline.pth
  • Clone umd-huang-lab/tracevla (checkout phi3 branch for TraceVLA-Phi3V), run pip install -e ., and install flash-attn==2.5.5
  • Load furonghuang-lab/tracevla_7b (or tracevla_phi3v) with AutoModelForCausalLM (trust_remote_code=True, bfloat16) and initialize TraceProcessor(cotracker_model_path)

[1] [3] [4]

Upstream-reported benchmarks

Upstream-reported only: Open Source Robotics has not run training, inference, or hardware evaluations for these benchmarks.

BenchmarkUpstream-reported scoreScope & caveats
SimplerEnv Fractal (Google Robot, Visual Matching & Variant Aggregation)TraceVLA-7B: 45.8% VM / 49.8% VA (vs OpenVLA-7B 36.5% VM / 35.8% VA); TraceVLA-Phi3V: 44.0% VM / 45.1% VA (vs OpenVLA-Phi3V 41.0% VM / 35.8% VA)Upstream-reported averages across 4 Fractal tasks (Pick up Coke, Move near, Open/Close Drawer, Put in Drawer); TraceVLA-Phi3V scores 0% on Put in Drawer; not independently reproduced by OSR.
SimplerEnv Bridge (WidowX, 4 tasks)TraceVLA-7B: 27.7% average (vs OpenVLA-7B 16.7%); TraceVLA-Phi3V: 21.9% average (vs OpenVLA-Phi3V 5.2%)Upstream-reported averages across Put Spoon, Put Carrot, Stack Block, Put Eggplant; TraceVLA-Phi3V scores 0% on Put Carrot; not independently reproduced by OSR.

[1] [3] [4]

Project notes

Two MIT checkpoints and Co-Tracker runtime coupling

Two ungated MIT-licensed checkpoints are published on Hugging Face: furonghuang-lab/tracevla_7b (7,541,239,360 BF16 parameters, 14.05 GB safetensors, fine-tuned from openvla/openvla-7b) and furonghuang-lab/tracevla_phi3v (7.72 GB PyTorch .bin, Phi3VForCausalLM with CLIP ViT-L/14-336, backed by the repository's separate phi3 branch). Both documented Transformers inference examples configure trust_remote_code=True and _attn_implementation='flash_attention_2', alongside an online TraceProcessor backed by Co-Tracker 3. In the TraceVLA-7B inference example, when Co-Tracker does not return a valid trace, the code falls back to duplicating the raw image ([image, image]) with the standard OpenVLA prompt; in the TraceVLA-Phi3V example, images are resized to 256x256 for Co-Tracker and 336x336 for the Phi-3-Vision encoder, and both the no-trace and trace branches use Phi3V chat-template prompts while passing [image, image_overlaid]. [1] [3] [4]

Upstream SimplerEnv results and documentation gaps

In the repository README and Hugging Face model cards, upstream reports SimplerEnv gains over base OpenVLA checkpoints: TraceVLA-7B reaches 45.8% Visual Matching and 49.8% Variant Aggregation on Fractal (versus 36.5% / 35.8% for OpenVLA-7B) and 27.7% average on Bridge (versus 16.7%), while TraceVLA-Phi3V reaches 44.0% VM / 45.1% VA on Fractal and 21.9% on Bridge (noting 0% on Fractal Put in Drawer and Bridge Put Carrot). Neither the README nor the model cards specify minimum inference VRAM, Co-Tracker latency overhead, or control frequency (Hz), and the documented training command assumes an 8-node × 8-GPU (64-GPU) cluster without a single-GPU LoRA recipe. All benchmark figures are upstream-reported; Open Source Robotics has not run training, fine-tuning, inference, or robot evaluations for this profile. [1] [3] [4]

What we checked

None. Prepared from public upstream sources and Hugging Face API metadata without hands-on training, fine-tuning, or inference.

Not yet established

  • Documented closed-loop control frequency (Hz)

Updates

Upstream last updated
Jan 8, 2025 [1]
Latest release
Jan 7, 2025 [3] [4]
Profile updated
Nov 21, 2026
Sources last checked
Nov 21, 2026
Next review
Feb 19, 2027
Reviewed revision
umd-huang-lab/tracevla@d5454af9ebe51e996005e13061ac3f6964e2923f; HF furonghuang-lab/tracevla_7b@23ac599e7fd95b1fb26dc9b93d03be4ad60d136f · combined hardware and software repository

Separate license matrix

MIT-licensed visual-trace-prompted VLA codebase and two ungated MIT checkpoints (7B OpenVLA and Phi-3-Vision); classified as partial openness because the visual-trace annotated training datasets remain unreleased and runtime requires external Co-Tracker 3 weights.

LayerAvailability / license
Codeavailable · MIT
Verified MIT License in root LICENSE file. [1] [2]
Model weightsavailable · MIT
Both furonghuang-lab/tracevla_7b and furonghuang-lab/tracevla_phi3v declare license: mit in HF YAML frontmatter; runtime also requires external facebook/cotracker3 weights. [3] [4]
Training datasetsunavailable · Upstream Open X-Embodiment is public (mixed licenses), but TraceVLA's visual-trace annotated Bridge and Fractal datasets remain unreleased ('[Coming soon]')
TraceVLA-specific visual-trace annotated datasets are marked '[Coming soon]' and not yet available. [1]
Documentationavailable · MIT
[1] [2]

Maintenance status, licensing, and unreleased trace data

Both the umd-huang-lab/tracevla codebase (verified in the root LICENSE file) and the two Hugging Face checkpoints declare the MIT License. However, the repository has had no commits since January 8, 2025, its README clone snippet still points to a non-existent openvla/tracevla URL, and the visual-trace annotated Bridge and Fractal training datasets remain marked [Coming soon]. Teams evaluating TraceVLA should also review the separate upstream terms for facebook/cotracker3 and the underlying openvla/openvla-7b (Llama-2) or Phi-3-vision base models. [1] [2] [3] [4]

References

Each source links to the exact revision we read. “Used for” shows where this page relies on it.

Correction guidance · Discussion is not enabled · Profile format 0.8.0-draft.1

Change history

2026-11-21 — Initial 0.8.0-draft.1 model profile prepared from umd-huang-lab/tracevla@d5454af9ebe5 and HF tracevla_7b + tracevla_phi3v checkpoints.