GraphEQA: 3Dセマンティックシーングラフを用いたリアルタイム身体性質問応答
GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answering
未知環境での身体性質問応答のため、リアルタイム3Dメトリックセマンティックシーングラフとタスク関連画像をマルチモーダルメモリとしてVLMに接地し、階層的計画で探索・回答する手法を提案。シミュレーションと実環境で成功率向上と計画ステップ削減を示した。
著者: Saumya Saxena, Blake Buchanan, Chris Paxton, Peiqi Liu, Bingqing Chen, Narunas Vaskevicius, Luigi Palmieri, Jonathan Francis, Oliver Kroemer
分類: cs.RO, cs.CL, cs.CV, cs.LG
原文アブストラクト
In Embodied Question Answering (EQA), agents must explore and develop a semantic understanding of an unseen environment to answer a situated question with confidence. This problem remains challenging in robotics, due to the difficulties in obtaining useful semantic representations, updating these representations online, and leveraging prior world knowledge for efficient planning and exploration. To address these limitations, we propose GraphEQA, a novel approach that utilizes real-time 3D metric-semantic scene graphs (3DSGs) and task relevant images as multi-modal memory for grounding Vision-Language Models (VLMs) to perform EQA tasks in unseen environments. We employ a hierarchical planning approach that exploits the hierarchical nature of 3DSGs for structured planning and semantics-guided exploration. We evaluate GraphEQA in simulation on two benchmark datasets, HM-EQA and OpenEQA, and demonstrate that it outperforms key baselines by completing EQA tasks with higher success rates and fewer planning steps. We further demonstrate GraphEQA in multiple real-world home and office environments.
