VideoMimic

Project notes

Begin with policy playback if you do not need new video data

This review uses commit cfded6d85dd55893074c844f74a49297e318980c, with no GitHub release listed. For simulation, follow simulation/setup.md: it recommends Ubuntu 18.04 or later, an NVIDIA GPU and driver 525 or later, then creates a Python 3.8 environment with PyTorch 2.3.1 and CUDA 12.1. Install Isaac Gym separately and install both videomimic_rl and videomimic_gym from the checkout. [3] [6]

The setup guide calls its environment rlgpu, while the workflow README calls it videomimic; activate the environment in which you actually installed those dependencies. From simulation/data, run bash download_videomimic_data.sh to obtain the linked checkpoints and data. From simulation, run bash videomimic_gym/legged_gym/scripts/play_terrain_policy.sh. The launcher selects a particular saved run and resources/data_config/4_motion.yaml, enables Viser and uses one simulated environment. The resulting browser viewer is a policy-playback route, not video reconstruction. [3] [4] [10]

Reconstructing your own video is a larger setup

The real-to-sim guide separates Python 3.12/CUDA 12.4+ preprocessing and retargeting from Python 3.10/CUDA 11.8 reconstruction. It lists Ubuntu 24.04 as its test environment and reports roughly 24 GB or more GPU memory for MegaSam on 300 frames. That figure is upstream guidance for one workload, not our measured minimum. [2]

There is a naming inconsistency to resolve before using the orchestration script: parts of the guide spell the reconstruction environment vm1reocn, but process_video.sh activates vm1recon. Make the installed environment name agree with the script. The guide also installs several dependencies from unpinned Git URLs and suggests a specific older PyRoki commit for changed variable names; record resolved dependency revisions instead of assuming current releases are interchangeable. [2] [8]

Download the reconstruction assets separately following real2sim/docs/directory.md; these include body models, checkpoints and robot assets. Final outputs include an OBJ scene mesh, PLY point clouds and HDF5 human/robot motion data. These files support a subsequent learning workflow; producing them does not establish successful physical control. [9]

Training and deployment remain research workflows

Upstream explicitly says not all four training stages were tested end-to-end before release. It also describes the sim-to-real code as not thoroughly tested or cleaned up. That path needs ROS 1 Noetic, elevation mapping, TorchScript checkpoints, CUDA and machine-specific path changes. Treat it as reference integration work with a defined robot test plan. [4] [5]

The root code license is MIT. External checkpoints, video datasets, body models and third-party components have separate rights that this review does not settle. External payloads were not acquired. We inspected the pinned setup and launch sources without installing, training or running a robot. [1] [2] [7] [9]