手なしでできる！一人称視点動画のエージェント・環境分解
Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videos
一人称視点動画から手（エージェント）と環境を分離する拡散モデルベースのインペインティング手法を提案し、物体検出や3D再構成、報酬・方策学習などのロボティクスタスクを容易にする。
著者: Matthew Chang, Aditya Prakash, Saurabh Gupta
分類: cs.CV, cs.LG, cs.RO
原文アブストラクト
The analysis and use of egocentric videos for robotic tasks is made challenging by occlusion due to the hand and the visual mismatch between the human hand and a robot end-effector. In this sense, the human hand presents a nuisance. However, often hands also provide a valuable signal, e.g. the hand pose may suggest what kind of object is being held. In this work, we propose to extract a factored representation of the scene that separates the agent (human hand) and the environment. This alleviates both occlusion and mismatch while preserving the signal, thereby easing the design of models for downstream robotics tasks. At the heart of this factorization is our proposed Video Inpainting via Diffusion Model (VIDM) that leverages both a prior on real-world images (through a large-scale pre-trained diffusion model) and the appearance of the object in earlier frames of the video (through attention). Our experiments demonstrate the effectiveness of VIDM at improving inpainting quality on egocentric videos and the power of our factored representation for numerous tasks: object detection, 3D reconstruction of manipulated objects, and learning of reward functions, policies, and affordances from videos.
