ロボセマンティックベンチ：VLAモデルの行動予測における意味的接地の診断
RoboSemanticBench: Diagnosing Semantic Grounding in Action Prediction for VLA Models
VLAモデルが複雑な指示の意味を理解して正しい物体を操作できるかを診断するベンチマークを提案し、多くのモデルが物体把握はできるが意味的に正しい選択がランダム以下であることを示した。
詳しい要約
1. どんなもの？
2. 先行研究と比べてどこがすごい？
3. 技術・手法の肝は？
4. どうやって有効だと検証した？
5. 議論はある？
6. 次に読むべき論文は？
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Bin Yu, Yao Zhang, Haishan Liu, Shijie Lian, Yuliang Wei, Xiaopeng Lin, Zhaolong Shen, Changti Wu, Ruina Hu, Bailing Wang, Cong Huang, Kai Chen
分類: cs.RO
原文アブストラクト
Vision-language-action (VLA) models are built on the premise that semantic understanding from pretrained language or vision-language backbones should guide robot action prediction. Yet robot fine-tuning is optimized as imitation over task-specific action distributions, and many evaluations can be solved through visual or instruction-action shortcuts. We introduce RoboSemanticBench (RSB), an embodied benchmark for diagnosing semantic grounding in action prediction: whether post-trained VLA models can use complex instruction semantics to select and manipulate the correct physical target. In each episode, a robot receives a multiple-choice math or general-knowledge question, observes candidate answer blocks, and must grasp the block corresponding to the correct answer. RSB covers controlled arithmetic, grade-school mathematical understanding, and commonsense or factual understanding under four-choice and ten-choice suites. Across representative VLA models, we find that many policies learn to grasp candidate blocks but select the semantically correct block at near-random or below-random rates after controlling for grasp success, revealing a persistent gap between backbone-level semantic competence and action prediction.
関連論文
- SurgVLA-Bench: 腹腔鏡手術ロボットのための視覚言語行動モデル評価ベンチマークVLA/評価ベンチマーク
- LIBERO-Para: VLAモデルの言い換え頑健性を評価する診断ベンチマークと指標VLA/評価ベンチマーク
