視覚的に行動とその効果を結びつける学習
Learning to Visually Connect Actions and their Effects
動画理解において、行動とその効果を視覚的に結びつける新しい概念CATEを提案し、行動選択と効果親和性評価の2側面を探求した。
著者: Paritosh Parmar, Eric Peh, Basura Fernando
分類: cs.CV, cs.AI, cs.LG, cs.RO
原文アブストラクト
We introduce the novel concept of visually Connecting Actions and Their Effects (CATE) in video understanding. CATE can have applications in areas like task planning and learning from demonstration. We identify and explore two different aspects of the concept of CATE: Action Selection (AS) and Effect-Affinity Assessment (EAA), where video understanding models connect actions and effects at semantic and fine-grained levels, respectively. We design various baseline models for AS and EAA. Despite the intuitive nature of the task, we observe that models struggle, and humans outperform them by a large margin. Our experiments show that in solving AS and EAA, models learn intuitive properties like object tracking and pose encoding without explicit supervision. We demonstrate that CATE can be an effective self-supervised task for learning video representations from unlabeled videos. The study aims to showcase the fundamental nature and versatility of CATE, with the hope of inspiring advanced formulations and models.
