MOSAIC: 対話的知覚によるロボット学習のための統合マルチセンサ物体特性表現の学習
MOSAIC: Learning Unified Multi-Sensory Object Property Representations for Robot Learning via Interactive Perception
視覚・触覚・聴覚の基盤モデルを蒸留・整合させ、ロボットが物体特性を統合的に理解する枠組みを提案。物体分類と物体把持タスクで有効性を示した。
著者: Gyan Tatiya, Jonathan Francis, Ho-Hsiang Wu, Yonatan Bisk, Jivko Sinapov
分類: cs.RO
原文アブストラクト
A holistic understanding of object properties across diverse sensory modalities (e.g., visual, audio, and haptic) is essential for tasks ranging from object categorization to complex manipulation. Drawing inspiration from cognitive science studies that emphasize the significance of multi-sensory integration in human perception, we introduce MOSAIC (Multimodal Object property learning with Self-Attention and Interactive Comprehension), a novel framework designed to facilitate the learning of unified multi-sensory object property representations. While it is undeniable that visual information plays a prominent role, we acknowledge that many fundamental object properties extend beyond the visual domain to encompass attributes like texture, mass distribution, or sounds, which significantly influence how we interact with objects. In MOSAIC, we leverage this profound insight by distilling knowledge from multimodal foundation models and aligning these representations not only across vision but also haptic and auditory sensory modalities. Through extensive experiments on a dataset where a humanoid robot interacts with 100 objects across 10 exploratory behaviors, we demonstrate the versatility of MOSAIC in two task families: object categorization and object-fetching tasks. Our results underscore the efficacy of MOSAIC's unified representations, showing competitive performance in category recognition through a simple linear probe setup and excelling in the fetch object task under zero-shot transfer conditions. This work pioneers the application of sensory grounding in foundation models for robotics, promising a significant leap in multi-sensory perception capabilities for autonomous systems. We have released the code, datasets, and additional results: https://github.com/gtatiya/MOSAIC.
関連論文
- BODYinTRANSIT-Lab:身体知覚と情動体験の感覚駆動型変化を調べるマルチモーダルデータセットマルチモーダル知覚
- UniOMA:最適輸送によるロボット知覚のための統合マルチモーダル構造アライメントマルチモーダル知覚
- ズームなしで詳細を見るv2:汎化可能なマルチモーダル知覚のための証拠連結マルチモーダル知覚
- SenseWeather: 極限環境下でのロボット知覚を支えるマルチモーダルセンシングマルチモーダル知覚
- ROMA: 実世界の物体中心マルチセンサ能動知覚のためのLLMシステムマルチモーダル知覚
- ReSPEC: 動的環境におけるオンライン多スペクトルセンサ再構成フレームワークマルチモーダル知覚
