CLIP-Fields: ロボット記憶のための弱教師あり意味場
CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory
CLIPやDeticなどWeb学習済みモデルのみを教師に使い、空間位置から意味埋め込みへの写像を学習する暗黙的シーン表現を提案。これをシーン記憶としてロボットが実環境で意味的ナビゲーションを行えることを示した。
著者: Nur Muhammad Mahi Shafiullah, Chris Paxton, Lerrel Pinto, Soumith Chintala, Arthur Szlam
分類: cs.RO, cs.CV
原文アブストラクト
We propose CLIP-Fields, an implicit scene model that can be used for a variety of tasks, such as segmentation, instance identification, semantic search over space, and view localization. CLIP-Fields learns a mapping from spatial locations to semantic embedding vectors. Importantly, we show that this mapping can be trained with supervision coming only from web-image and web-text trained models such as CLIP, Detic, and Sentence-BERT; and thus uses no direct human supervision. When compared to baselines like Mask-RCNN, our method outperforms on few-shot instance identification or semantic segmentation on the HM3D dataset with only a fraction of the examples. Finally, we show that using CLIP-Fields as a scene memory, robots can perform semantic navigation in real-world environments. Our code and demonstration videos are available here: https://mahis.life/clip-fields
