一人称RGB動画からの3D手姿勢推定と行動認識のための階層的時間Transformer
Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videos
一人称視点のRGB動画から、短期的な時間情報で手の3D姿勢を推定し、長期的な時間情報で行動を認識する2段構成のTransformerフレームワークを提案した論文。
著者: Yilin Wen, Hao Pan, Lei Yang, Jia Pan, Taku Komura, Wenping Wang
分類: cs.CV, cs.RO
原文アブストラクト
Understanding dynamic hand motions and actions from egocentric RGB videos is a fundamental yet challenging task due to self-occlusion and ambiguity. To address occlusion and ambiguity, we develop a transformer-based framework to exploit temporal information for robust estimation. Noticing the different temporal granularity of and the semantic correlation between hand pose estimation and action recognition, we build a network hierarchy with two cascaded transformer encoders, where the first one exploits the short-term temporal cue for hand pose estimation, and the latter aggregates per-frame pose and object information over a longer time span to recognize the action. Our approach achieves competitive results on two first-person hand action benchmarks, namely FPHA and H2O. Extensive ablation studies verify our design choices.
