音声視覚ナビゲーションにおける汎化可能な音声表現に向けて
Towards Generalisable Audio Representations for Audio-Visual Navigation
音声視覚ナビゲーションで未知の音への汎化性能を高めるため、対照学習で音声エンコーダを正則化し、音に依存しない目標駆動潜在表現を学習する手法を提案。既存フレームワークに組み込むだけで性能が向上する。
著者: Shunqi Mao, Chaoyi Zhang, Heng Wang, Weidong Cai
分類: cs.SD, cs.CV, cs.LG, cs.RO, eess.AS
原文アブストラクト
In audio-visual navigation (AVN), an intelligent agent needs to navigate to a constantly sound-making object in complex 3D environments based on its audio and visual perceptions. While existing methods attempt to improve the navigation performance with preciously designed path planning or intricate task settings, none has improved the model generalisation on unheard sounds with task settings unchanged. We thus propose a contrastive learning-based method to tackle this challenge by regularising the audio encoder, where the sound-agnostic goal-driven latent representations can be learnt from various audio signals of different classes. In addition, we consider two data augmentation strategies to enrich the training sounds. We demonstrate that our designs can be easily equipped to existing AVN frameworks to obtain an immediate performance gain (13.4%$\uparrow$ in SPL on Replica and 12.2%$\uparrow$ in SPL on MP3D). Our project is available at https://AV-GeN.github.io/.
