音声から視覚に接地した言語を学ぶ:深層音響表現による新手法
Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speech
視覚知覚と生の音声入力を用いて、接地言語獲得が可能であることを示した研究。自己教師あり音声表現を活用し、テキスト入力への依存を減らしつつ、特定集団への包摂性を高める。
著者: Gaoussou Youssouf Kebe, Luke E. Richards, Edward Raff, Francis Ferraro, Cynthia Matuszek
分類: cs.CL, cs.AI, cs.LG, cs.RO
原文アブストラクト
Learning to understand grounded language, which connects natural language to percepts, is a critical research area. Prior work in grounded language acquisition has focused primarily on textual inputs. In this work we demonstrate the feasibility of performing grounded language acquisition on paired visual percepts and raw speech inputs. This will allow interactions in which language about novel tasks and environments is learned from end users, reducing dependence on textual inputs and potentially mitigating the effects of demographic bias found in widely available speech recognition systems. We leverage recent work in self-supervised speech representation models and show that learned representations of speech can make language grounding systems more inclusive towards specific groups while maintaining or even increasing general performance.
