EgoSafetyBench：具身VLMをランタイム安全監視役として評価する診断用エゴセントリック動画ベンチマーク
EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards
ロボット視点の動画で、安全監視VLMが本当の危険を見逃さず、見かけだけの異常に過剰介入しないかを診断するベンチマークを構築し、10種のVLMを評価した。
詳しい要約
1. どんなもの？
2. 先行研究と比べてどこがすごい？
3. 技術・手法の肝は？
4. どうやって有効だと検証した？
5. 議論はある？
6. 次に読むべき論文は？
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Siddhant Panpatil, Arth Singh, Mijin Koo, Chaeyun Kim, Haon Park, Dasol Choi
分類: cs.CV, cs.AI
原文アブストラクト
Vision-language models (VLMs) are now proposed as runtime safety guards for embodied agents in homes and factories. A deployable guard must catch genuinely unsafe situations while avoiding unnecessary intervention on routine but superficially alarming activity, a distinction that binary safety benchmarks obscure. We introduce EgoSafetyBench, an egocentric video benchmark of 1,200 robot-view scenarios annotated at half-second granularity, to evaluate VLMs as streaming guards across two tracks. The situational track (800 scenarios) spans four families, from routine and safe-but-suspicious scenes to obvious and contextual hazards. The visual-channel track (400 scenarios) targets in-scene text-a sign, sticker, or label visible in the scene-that can misrepresent the physical situation, pairing each misleading sign with a truthful version to test both whether a guard flags the text as misleading and whether the text corrupts its physical-safety judgment. Both tracks use contrastive ladders: near-identical scenarios differing only in a single visible deciding cue, so a correct call must hinge on that cue rather than the overall scene type. We evaluate ten open- and closed-source VLMs. We find that while guards reliably recognize videos containing hazards, they often miss specific hazardous moments, particularly contextual hazards. Furthermore, misleading in-scene signs degrade all tested guards: vulnerable models miss up to a third of hazards, while robust models over-intervene on safe content. Matched controls reveal that apparent safety robustness often reflects indiscriminate alarming rather than true physical reasoning.
関連論文
- SafeVLA-Bench: 視覚言語行動モデルにおける成功と安全性のギャップを評価するベンチマークVLA/安全性評価
- いつ、どの程度深刻に：運転VLAのためのシナリオ別安全エンベロープVLA/安全性評価
- LIBERO-Safety: 視覚言語行動モデルにおける物理的・意味的安全性の包括的ベンチマークVLA/安全性評価
- ForesightSafety-VLA: 視覚・言語・行動モデルのための統合診断安全ベンチマークVLA/安全性評価
- 視覚言語モデルにおける衝突接地の探求：安全な人間ロボット協調のためにVLA/安全性評価
- HazardArena: 視覚言語行動モデルにおける意味的安全性の評価VLA/安全性評価
