Vid2WAM: Distilling Video Diffusion Priors into World Action Models

Chenhao Qiu^1,*, Ruixiang Wang^2,*,†, Runyi Zhao^2,*, Sixu Lin², Songen Gu¹, Shufeng Nan¹, Guiliang Liu², Kui Jia², Yanwei Fu^1,3, Simo Wu^1,‡

¹ Fudan University ² The Chinese University of Hong Kong, Shenzhen ³ Shanghai Innovation Institute

^* Equal contribution. ^† Project lead. ^‡ Corresponding author.

Vid2WAM training architecture with a frozen video teacher and inverse dynamics model supervising a trainable student, followed by student-only inference.

Teacher-generated future latents and IDM-derived pseudo-actions complement real demonstrations during training. At inference time, the video teacher and IDM are discarded, leaving only the student action policy.

Video-Foundation-Model-to-WAM Distillation: We rethink the source of future supervision for World Action Models and formulate a video-foundation-model-to-WAM distillation framework, using generated futures from video foundation models as direct supervision beyond recorded target-task expert trajectories.
Dual-Channel, Source-Aware Transfer: We introduce a dual-channel, source-aware transfer strategy that jointly distills future visual dynamics and IDM-inferred actions, with residual adapters reducing interference from noisy pseudo-actions.
Generalization with Efficient Inference: We demonstrate in simulation and real-world experiments that Vid2WAM improves expert-demonstration efficiency and novel-task generalization, while retaining efficient student-only inference.

Abstract

World Action Models (WAMs) improve robot policy learning by jointly modeling future visual dynamics and actions. However, their scalability and generalization remain constrained by their reliance on costly expert demonstrations. We challenge this by asking whether future supervision for WAMs must originate from target-task expert trajectories. In this paper, we propose Vid2WAM, an offline distillation framework that transfers visual diffusion priors from a large video foundation model into a compact WAM student. Given an observation and language instruction, Vid2WAM distills supervision through two complementary channels: task-conditioned future rollouts directly supervise the student's future prediction branch, while an inverse dynamics model recovers embodiment-specific pseudo-actions for action learning. To robustly integrate synthetic and real supervision, we introduce source-aware residual action adaptation that learns source-specific corrections around a shared action backbone and mitigates interference from noisy pseudo-actions. During inference, both the video teacher and inverse dynamics model are discarded, leaving only the WAM student for efficient deployment. Simulation and real-world experiments demonstrate that Vid2WAM improves novel-task generalization and data efficiency under limited expert demonstrations while preserving low-latency inference.

Simulation Results

We consider two complementary evaluation regimes. In the low-data regime, each task provides only 10 expert trajectories, with Vid2WAM receiving an equal number of offline teacher-generated trajectories. In the novel-task regime, we withhold all expert trajectories from selected target tasks: baselines are directly transferred from the remaining tasks, whereas Vid2WAM uses only target-task instructions and initial observations to generate pseudo supervision. These regimes respectively evaluate policy learning under scarce real demonstrations and adaptation to novel tasks without target-task expert supervision. All charts below report success rate in percent; higher is better.

π_0.5 Motus Fast-WAM Vid2WAM

RoboTwin 2.0 · Novel Tasks

15 unseen tasks only

46.5

48.9

45.0

54.7

Clean

46.3

51.5

42.8

55.3

Randomized

Vid2WAM improves over Fast-WAM by 9.7 and 12.5 points on unseen tasks.

LIBERO · Average Success

Four task suites

86.9

87.7

87.3

89.7

Low-data

77.9

72.1

76.7

78.3

Novel

Vid2WAM achieves the best average result in both evaluation regimes.

LIBERO-Plus · Robustness

Overall across seven perturbations

27.9

33.6

39.0

Low-data

41.9

45.8

48.1

Novel

Vid2WAM remains strongest under camera, robot, language, lighting, and scene perturbations.

Real Robot · Novel Tasks

20 trials per task

Pick test tube

Take tissue

Tomato basket

Novel tasks receive no target-task real expert trajectories.

Inference Latency

Milliseconds per action chunk · RTX 4090

π_0.576.7 ms

Fast-WAM209.1 ms

Vid2WAM212.5 ms

Motus2,414.9 ms

Teacher Policy4,894.5 ms

Offline distillation preserves Fast-WAM-level latency and is about 23× faster than online teacher control.

Qualitative Future Predictions

From identical initial observations and language instructions, Vid2WAM predicts task completion beyond intermediate grasping or opening steps.

Future predictions from Fast-WAM and Vid2WAM for two representative novel manipulation tasks. — Fast-WAM (top) and Vid2WAM (bottom) future predictions for representative novel tasks.

Teacher-Generated Rollouts

Six representative rollouts cover LIBERO, RoboTwin, and real-world settings across low-data, novel-task, clean, and randomized conditions.

Six teacher-generated rollout examples from LIBERO, RoboTwin, and real-world tasks, each shown as six temporally ordered keyframes. — Each row presents six temporally ordered keyframes conditioned on the displayed language instruction.

Real-World Experiments

Vid2WAM is evaluated on nine contact-rich bimanual tasks, including novel object-level and behavior-level tasks without target-task expert trajectories.

Vid2WAM rollouts on representative seen and novel real-world tasks. — Representative seen and novel real-world executions. Frames progress from left to right.

Video demonstrations of all real-world tasks, comparing Vid2WAM with Fast-WAM.

Citation

If you find our work helpful, please cite:

@misc{qiu2026vid2wamdistillingvideodiffusion,
  title={Vid2WAM: Distilling Video Diffusion Priors into World Action Models},
  author={Chenhao Qiu and Ruixiang Wang and Runyi Zhao and Sixu Lin and Songen Gu and Shufeng Nan and Guiliang Liu and Kui Jia and Yanwei Fu and Simo Wu},
  year={2026},
  eprint={2608.08558},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2608.08558},
}