PROGRESSOR: 自己教師ありオンライン精緻化を伴う知覚誘導型報酬推定器
PROGRESSOR: A Perceptually Guided Reward Estimator with Self-Supervised Online Refinement
大規模な一人称視点動画からタスク非依存の報酬関数を自己教師ありで学習し、オンラインRL中に敵対的に精緻化することで、人手による報酬設計なしにロボットの複雑な行動獲得を可能にするフレームワーク。
著者: Tewodros Ayalew, Xiao Zhang, Kevin Yuanbo Wu, Tianchong Jiang, Michael Maire, Matthew R. Walter
分類: cs.RO, cs.AI
原文アブストラクト
We present PROGRESSOR, a novel framework that learns a task-agnostic reward function from videos, enabling policy training through goal-conditioned reinforcement learning (RL) without manual supervision. Underlying this reward is an estimate of the distribution over task progress as a function of the current, initial, and goal observations that is learned in a self-supervised fashion. Crucially, PROGRESSOR refines rewards adversarially during online RL training by pushing back predictions for out-of-distribution observations, to mitigate distribution shift inherent in non-expert observations. Utilizing this progress prediction as a dense reward together with an adversarial push-back, we show that PROGRESSOR enables robots to learn complex behaviors without any external supervision. Pretrained on large-scale egocentric human video from EPIC-KITCHENS, PROGRESSOR requires no fine-tuning on in-domain task-specific data for generalization to real-robot offline RL under noisy demonstrations, outperforming contemporary methods that provide dense visual reward for robotic learning. Our findings highlight the potential of PROGRESSOR for scalable robotic applications where direct action labels and task-specific rewards are not readily available.
