大規模（視覚）言語モデルを活用したロボットの3Dシーン理解
Leveraging Large (Visual) Language Models for Robot 3D Scene Understanding
事前学習済み言語モデルと視覚言語モデルを用いて、ロボットの3Dシーン理解（部屋分類）を行い、言語のみ・視覚言語併用の複数手法を比較した。最良手法は約70%の部屋分類精度を達成し、純視覚やグラフ分類器を上回り、言語に由来する汎化・転移性能も示した。
著者: William Chen, Siyi Hu, Rajat Talak, Luca Carlone
分類: cs.RO, cs.CL, cs.CV, cs.LG
原文アブストラクト
Abstract semantic 3D scene understanding is a problem of critical importance in robotics. As robots still lack the common-sense knowledge about household objects and locations of an average human, we investigate the use of pre-trained language models to impart common sense for scene understanding. We introduce and compare a wide range of scene classification paradigms that leverage language only (zero-shot, embedding-based, and structured-language) or vision and language (zero-shot and fine-tuned). We find that the best approaches in both categories yield $\sim 70\%$ room classification accuracy, exceeding the performance of pure-vision and graph classifiers. We also find such methods demonstrate notable generalization and transfer capabilities stemming from their use of language.
