SORT3D: 大規模言語モデルを用いたゼロショット3Dグラウンディングのための空間オブジェクト中心推論ツールボックス
SORT3D: Spatial Object-centric Reasoning Toolbox for Zero-Shot 3D Grounding Using Large Language Models
2Dの物体属性とヒューリスティックな空間推論ツールボックスをLLMの逐次推論能力と組み合わせ、テキスト-3Dデータなしで未知環境にゼロショット適用できる3Dグラウンディング手法を提案。
著者: Nader Zantout, Haochen Zhang, Pujith Kachana, Jinkai Qiu, Guofei Chen, Ji Zhang, Wenshan Wang
分類: cs.CV, cs.AI, cs.RO
原文アブストラクト
Interpreting object-referential language and grounding objects in 3D with spatial relations and attributes is essential for robots operating alongside humans. However, this task is often challenging due to the diversity of scenes, large number of fine-grained objects, and complex free-form nature of language references. Furthermore, in the 3D domain, obtaining large amounts of natural language training data is difficult. Thus, it is important for methods to learn from little data and zero-shot generalize to new environments. To address these challenges, we propose SORT3D, an approach that utilizes rich object attributes from 2D data and merges a heuristics-based spatial reasoning toolbox with the ability of large language models (LLMs) to perform sequential reasoning. Importantly, our method does not require text-to-3D data for training and can be applied zero-shot to unseen environments. We show that SORT3D achieves state-of-the-art zero-shot performance on complex view-dependent grounding tasks on two benchmarks. We also implement the pipeline to run real-time on two autonomous vehicles and demonstrate that our approach can be used for object-goal navigation on previously unseen real-world environments. All source code for the system pipeline is publicly released at https://github.com/nzantout/SORT3D.
関連論文
- CitySTAR: オープンボキャブラリ都市3Dグラウンディングのための構造的・トポロジー認識推論3Dグラウンディング
- 3EED: あらゆる場所のあらゆるものを3Dでグラウンディング3Dグラウンディング
- 粗から細へのアプローチによるマルチモーダル3D占有グラウンディング3Dグラウンディング
