VPP2 releases training code for robot visual history
Robotera’s VPP2 project released training code on October 12 for a video model that uses earlier observations on RoboDojo. Researchers now have a documented route from its released video checkpoint into a robot policy that uses visual history.
The release extends research first submitted on October 7 and revised on October 8. The VPP2 paper describes learning from manipulation videos before adding an action model that converts visual predictions into robot movements.
That order matters to the authors’ approach. They argue that ordinary video generators can predict the wrong movement for a manipulation instruction, while adding action training can damage useful visual representations. Their method starts with detailed descriptions of complete manipulation events, then adapts the predictor for shorter planning intervals and trains an action expert.
Why earlier observations matter
The RoboDojo benchmark paper gives a concrete example. A robot sees an object disappear on a conveyor and must later select another object of the same category. Another task asks it to remember and repeat a sorting order demonstrated by a different arm. A single current image cannot always contain the information needed to choose the right action.
The new training recipe starts from VPP2’s Stage 2 video checkpoint. Each training window combines the episode’s first image, seven earlier observations, the current observation and sixteen future frames. The video model trains for 10,000 steps before initialization of the action expert.
The video stage uses all 3,500 demonstrations and a global batch of 832. Local policy evaluation uses two GPUs and paired weights totaling about 36.85 GB, plus shared encoders. Linux, compatible NVIDIA hardware and RoboDojo’s simulator assets are required.
RoboDojo evaluates 42 simulation tasks across generalization, memory, precision, extended task execution and instruction following. Its protocol runs 50 episodes per task and averages the five capability dimensions, rather than letting larger categories dominate. The separate physical benchmark contains 18 tasks. Those physical tests examine deployment conditions that simulation cannot fully capture.
What the evidence establishes
In its earlier paper, the VPP2 team reports 58.5 percent average success across ten categories on an ALOHA robot, without tuning for each test task. The robot embodiment had appeared during training. The paper separately reports 32.26 percent success on RoboDojo simulation. These are different evaluations, and both remain results reported by the developers.
The October 12 addition makes an intermediate training stage inspectable. It does not establish a new performance gain or show that the policy will transfer reliably to arbitrary robots and environments.
The repository licenses its code under MIT. Model weights, datasets and simulator assets retain separate terms that users must check.
Our NAVA WAM report describes another approach to learning robot action policies from video.
