動的音声視覚ナビゲーションのための二段階マルチモーダル学習:物理的整合からシーン集約へ
Two-Stage Multi-modal Learning for Dynamic Audio-Visual Navigation: From Physical Alignment to Scene Aggregation
移動する音源を視覚と聴覚で追跡する動的音声視覚ナビゲーションにおいて、音強度と視覚形状を物理的に整合させる段階とTransformerでシーンを集約する段階からなる二段階学習法を提案し、未知音カテゴリへの汎化とナビゲーション効率を改善した。
詳しい要約
1. どんなもの?
2. 先行研究と比べてどこがすごい?
3. 技術・手法の肝は?
4. どうやって有効だと検証した?
5. 議論はある?
6. 次に読むべき論文は?
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Jia Li, Yinfeng Yu
分類: cs.SD, cs.AI, cs.RO, eess.AS, eess.SP
原文アブストラクト
Embodied Audio-Visual Navigation (AVN) requires an agent to locate sound targets by jointly leveraging visual and auditory cues. While prior work has achieved notable success with static sound sources, real-world scenarios often involve moving targets, giving rise to the more challenging problem of \textbf{Dynamic Audio-Visual Navigation (D-AVN)}. Existing approaches largely rely on semantic auditory features, which limits generalization to unheard sound categories and hinders fine-grained spatial reasoning. To address these challenges, we propose a novel \textbf{Two-Stage Multi-modal Learning} framework for D-AVN. In the Physical Alignment stage, the proposed \textbf{Intensity-Calibrated Geometry Encoder (ICGE)} performs physics-informed alignment of sound intensity with visual geometry to capture category-agnostic spatial correspondence. In the Scene Aggregation stage, a Transformer-based \textbf{Cross-Modal Scene Aggregator (CMSA)} dynamically integrates multi-modal features to construct a robust state representation for policy learning. Extensive experiments on Replica and Matterport3D demonstrate that our method achieves state-of-the-art success rates, while significantly improving generalization and navigation efficiency in complex environments and under unheard sound scenarios.
関連論文
- 移動ロボットの状況依存マルチモーダル制御:仮想環境を通じたナビゲーションマルチモーダルナビゲーション
