離散ジェスチャトークン学習による音声同期ジェスチャ生成
Co-Speech Gesture Synthesis using Discrete Gesture Token Learning
音声に合わせたジェスチャ動作を、離散トークン化と自己回帰トランスフォーマで生成する2段階手法を提案し、多様で自然な動作を実現した。
著者: Shuhong Lu, Youngwoo Yoon, Andrew Feng
分類: cs.CV, cs.AI, cs.LG, cs.RO
原文アブストラクト
Synthesizing realistic co-speech gestures is an important and yet unsolved problem for creating believable motions that can drive a humanoid robot to interact and communicate with human users. Such capability will improve the impressions of the robots by human users and will find applications in education, training, and medical services. One challenge in learning the co-speech gesture model is that there may be multiple viable gesture motions for the same speech utterance. The deterministic regression methods can not resolve the conflicting samples and may produce over-smoothed or damped motions. We proposed a two-stage model to address this uncertainty issue in gesture synthesis by modeling the gesture segments as discrete latent codes. Our method utilizes RQ-VAE in the first stage to learn a discrete codebook consisting of gesture tokens from training data. In the second stage, a two-level autoregressive transformer model is used to learn the prior distribution of residual codes conditioned on input speech context. Since the inference is formulated as token sampling, multiple gesture sequences could be generated given the same speech input using top-k sampling. The quantitative results and the user study showed the proposed method outperforms the previous methods and is able to generate realistic and diverse gesture motions.
