PASA-KV

World action model security · Visual patch attack

Attacking Action-Facing
Visual Routes

PASA-KV investigates how a small RGB patch can disrupt the visual key/value routes used by robot actions in world action models (WAM)

Zhongbo Wu1,2,3, Jiakai Huang1,3,*, Zheng Zhang3,4, Siyu Xu5, Kaixiang Chen1,*, Xiaohong Liu1, Shan You3, Chang Xu5, Tao Huang1,†

1 Shanghai Jiao Tong University 2 Shanghai Innovation Institute 3 ACE Robotics 4 Nanyang Technological University 5 School of Computer Science, The University of Sydney

* Research intern   † Corresponding author

Paper Code Coming soonVideos
Shanghai Jiao Tong University Shanghai Innovation Institute ACE Robotics Nanyang Technological University University of Sydney

Abstract

PASA-KV is a white box visual patch attack on world action models. It first uses short probes to identify visual key/value routes that influence action readouts, then selects layers with Tail-AV on held-out clean and patched observations. A fresh RGB patch is trained against the selected routes while the model remains frozen. We evaluate the attack through closed loop policy success in simulation and on a real robot.

Method

The optimized object is an RGB patch on camera pixels. Visual keys affect where action queries read; visual values carry the content they receive

PASA-KV method figure: the visual key and value attack surface, action readout, and the probe, select, train, evaluate process
Method overview. PASA-KV probes candidate layers, selects action-facing routes using Tail-AV, and trains a fresh patch for evaluation

Simulation results

Policy success rate measures completed tasks under closed loop evaluation. Lower success under attack indicates a stronger effect

Main policy success results for FastWAM, LingBot-VA, and Cosmos Policy
Main results. Policy success on LIBERO-10 Eval500 across three world action models

FastWAM · Eval500 Clean policy success was 92.4%; PASA-KV Top-5 reduced it to 0.0% across 500 evaluation episodes

Layer selection · Eval200 Tail-AV selected layers yielded 1.5% policy success, compared with 47.5% when layers were selected by probe K/V displacement under matched conditions

Detailed simulation results and evaluation scope ↗

Real robot evaluation

Piper robot demonstrations from clean and PASA-KV Top-5 conditions, shown at 5× speed

Clean FastWAM · Piper pen and cup task Open video ↗
PASA-KV Top-5 · Piper pen and cup task Open video ↗
Piper robot task success under clean and attacked conditions across three tasks
Real robot results. Success counts for three tasks with nonzero clean success, with 10 trials per condition. The videos above show qualitative examples

Detailed Piper results and trial counts ↗