Verify2Act: Critic-Guided Latent World Models for Verifying Language-Conditioned Manipulation Plans

A vision-language model proposes candidate plans, a latent world model imagines each one, and a critic checks the imagined outcome against the goal before the robot moves.

Overview

Abstract

Vision-Language Models (VLMs) can propose multi-step manipulation plans, but they have no grounded model of how a scene will change, and the pixel-level world models that could check those plans are slow and drift over long horizons. We present Verify2Act, a modular verification framework that separates semantic action proposal from latent physical verification. A VLM proposes candidate plans, a latent world model (V2A-WM) imagines each of them in the DINOv2 feature space, and a dual-head critic checks the imagined rollouts for temporal consistency and goal satisfaction before anything is executed. V2A-WM extends recent flow-matching world models with cross-attention action grounding and a causal temporal history window. We deploy the full system on a low-cost robot arm, training the world model and critic only on data generated in a digital twin. Across six language-conditioned tasks, Verify2Act reaches 83.3% task success, compared with 61.7% for a latent world model without our architectural changes, 53.3% for the VLM alone, and 46.7% for a pixel-diffusion world model. It verified every plan it executed and incurred no planning-induced failures: all of its failed episodes were caused by low-level grasp or placement slips. In simulation, Verify2Act obtains the highest five-task chain success on CALVIN among the compared baselines with 30× fewer reflective VLM calls than pixel-diffusion verification, and the highest success on cluttered nut assembly.

Method

Verify2Act treats the VLM only as a source of candidate plans. Each candidate is rolled out by V2A-WM in the frozen DINOv2 latent space, and a dual-head critic scores the imagined rollout: one head checks that consecutive imagined states are temporally consistent, the other scores the final imagined state against the language goal. Plans that fail are sent back to the VLM for re-planning; only an accepted plan reaches the robot.

V2A-WM

V2A-WM builds on the flow-matching cores of DINO-WM and RLA-WM with two changes: cross-attention action grounding between DINOv2 patch features and CLIP tokens, which ties language commands to visual object patches, and a causal temporal history window with validity masks, which preserves visual memory across gripper occlusions.

Sim-to-real through a digital twin

For the robot, the world model and the critic are trained only on data generated in a digital twin of the workspace (20k episodes). At test time, a real2sim step aligns the camera frame with the twin before the plan is imagined.

Real-World Execution

A Yahboom DOFBOT with an arm-mounted camera and four colored blocks. Robot footage is sped up 4× or 6×, as labeled in each clip.

More rollouts

The bin-clearing task on two other layouts, unedited apart from the speed-up.

These demonstrations were filmed without the letter-size sheet used in the paper's evaluation. The planning panels were computed from the recorded camera frames and the plans the VLM proposed during the filmed runs.

Results

Real robot

All methods share the same VLM (GPT-4o), prompts and low-level skills, and differ only in how a proposed plan is verified.

Simulation: CALVIN

Simulation: cluttered nut assembly

Citation

@misc{eze2026verify2act,
  title  = {Verify2Act: Critic-Guided Latent World Models for Verifying
            Language-Conditioned Manipulation Plans},
  author = {Eze, Chrisantus and Crick, Christopher},
  year   = {2026},
  note   = {Under review}
}