日常の一人称視点画像における3D手姿勢推定
3D Hand Pose Estimation in Everyday Egocentric Images
日常の一人称視点画像から3D手姿勢を推定する課題に対し、クロップ入力やカメラ情報、補助監督、データ拡大の影響を分析し、WildHandsシステムを提案。4つの多様なデータセットでゼロショット評価を行い、既存手法を大幅に上回る性能を達成した。
著者: Aditya Prakash, Ruisen Tu, Matthew Chang, Saurabh Gupta
分類: cs.CV, cs.AI, cs.LG, cs.RO
原文アブストラクト
3D hand pose estimation in everyday egocentric images is challenging for several reasons: poor visual signal (occlusion from the object of interaction, low resolution & motion blur), large perspective distortion (hands are close to the camera), and lack of 3D annotations outside of controlled settings. While existing methods often use hand crops as input to focus on fine-grained visual information to deal with poor visual signal, the challenges arising from perspective distortion and lack of 3D annotations in the wild have not been systematically studied. We focus on this gap and explore the impact of different practices, i.e. crops as input, incorporating camera information, auxiliary supervision, scaling up datasets. We provide several insights that are applicable to both convolutional and transformer models leading to better performance. Based on our findings, we also present WildHands, a system for 3D hand pose estimation in everyday egocentric images. Zero-shot evaluation on 4 diverse datasets (H2O, AssemblyHands, Epic-Kitchens, Ego-Exo4D) demonstrate the effectiveness of our approach across 2D and 3D metrics, where we beat past methods by 7.4% - 66%. In system level comparisons, WildHands achieves the best 3D hand pose on ARCTIC egocentric split, outperforms FrankMocap across all metrics and HaMeR on 3 out of 6 metrics while being 10x smaller and trained on 5x less data.
