視覚的手がかりが二者間対話の予測的ターンテイキングを向上させる
Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interaction
表情・頭部姿勢・視線などの視覚情報を音声と組み合わせたマルチモーダルなターンテイキング予測モデルMM-VAPを提案し、音声のみの手法を上回る精度を達成した。
詳しい要約
1. どんなもの？
2. 先行研究と比べてどこがすごい？
3. 技術・手法の肝は？
4. どうやって有効だと検証した？
5. 議論はある？
6. 次に読むべき論文は？
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Sam O'Connor Russell, Naomi Harte
分類: cs.CL, cs.RO
原文アブストラクト
Turn-taking is richly multimodal. Predictive turn-taking models (PTTMs) facilitate naturalistic human-robot interaction, yet most rely solely on speech. We introduce MM-VAP, a multimodal PTTM which combines speech with visual cues including facial expression, head pose and gaze. We find that it outperforms the state-of-the-art audio-only in videoconferencing interactions (84% vs. 79% hold/shift prediction accuracy). Unlike prior work which aggregates all holds and shifts, we group by duration of silence between turns. This reveals that through the inclusion of visual features, MM-VAP outperforms a state-of-the-art audio-only turn-taking model across all durations of speaker transitions. We conduct a detailed ablation study, which reveals that facial expression features contribute the most to model performance. Thus, our working hypothesis is that when interlocutors can see one another, visual cues are vital for turn-taking and must therefore be included for accurate turn-taking prediction. We additionally validate the suitability of automatic speech alignment for PTTM training using telephone speech. This work represents the first comprehensive analysis of multimodal PTTMs. We discuss implications for future work and make all code publicly available.
関連論文
- XBot-Omni：身体理解と対話のためのオムニモーダルモデルマルチモーダル対話
- 談話一貫性に基づく能動学習型マルチモーダル安全対話システムマルチモーダル対話
- マルチモーダル対話エージェントを構築するモジュラーアーキテクチャマルチモーダル対話
