音声視覚ナビゲーションのための残差クロスモーダル融合ネットワーク
Residual Cross-Modal Fusion Networks for Audio-Visual Navigation
音声と視覚の双方向残差融合により、異種モダリティ間の相補的な統合と微細な位置合わせを実現し、未知環境での音源探索性能とクロスドメイン汎化を向上させた研究。
詳しい要約
1. どんなもの？
2. 先行研究と比べてどこがすごい？
3. 技術・手法の肝は？
4. どうやって有効だと検証した？
5. 議論はある？
6. 次に読むべき論文は？
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Yi Wang, Yinfeng Yu, Bin Ren
分類: cs.CV, cs.AI, cs.RO
原文アブストラクト
Audio-visual embodied navigation aims to enable an agent to autonomously localize and reach a sound source in unseen 3D environments by leveraging auditory cues. The key challenge of this task lies in effectively modeling the interaction between heterogeneous features during multimodal fusion, so as to avoid single-modality dominance or information degradation, particularly in cross-domain scenarios. To address this, we propose a Cross-Modal Residual Fusion Network, which introduces bidirectional residual interactions between audio and visual streams to achieve complementary modeling and fine-grained alignment, while maintaining the independence of their representations. Unlike conventional methods that rely on simple concatenation or attention gating, CRFN explicitly models cross-modal interactions via residual connections and incorporates stabilization techniques to improve convergence and robustness. Experiments on the Replica and Matterport3D datasets demonstrate that CRFN significantly outperforms state-of-the-art fusion baselines and achieves stronger cross-domain generalization. Notably, our experiments also reveal that agents exhibit differentiated modality dependence across different datasets. The discovery of this phenomenon provides a new perspective for understanding the cross-modal collaboration mechanism of embodied agents.
関連論文
- 知識駆動型シーンプライアによる意味的音声視覚身体ナビゲーション音声視覚ナビゲーション
- 音響攻撃に頑健な音声視覚ナビゲーション音声視覚ナビゲーション
- 動的音声視覚ナビゲーション：未知の3D環境で未聴の移動音源を捕まえる音声視覚ナビゲーション
