行動ラベルから集合へ：訂正フィードバックを用いた模倣学習における行動監督の再考
From Action Labels to Sets: Rethinking Action Supervision for Imitation Learning from Corrective Feedback
人間の訂正フィードバックを活用し、単一の行動ラベルではなく望ましい行動の集合に対して方策を学習するCLICを提案。正確なデータでは従来手法と同等、ノイズや相対・部分的なフィードバック下で大幅に頑健であることを示した。
著者: Zhaoting Li, Rodrigo Pérez-Dattari, Robert Babuska, Cosimo Della Santina, Jens Kober
分類: cs.RO
原文アブストラクト
Behavior cloning (BC) optimizes policies by treating human demonstrations as pointwise action labels. While effective with accurate action labels, this formulation is brittle in practice: when human-provided actions are imperfect, treating each label as an exact target can steer the policy away from the underlying desired behavior, particularly when expressive models are used (e.g., energy-based models). As a result, we propose a human-in-the-loop alternative that replaces pointwise supervision with set-valued action targets. We introduce Contrastive policy Learning from Interactive Corrections (CLIC). CLIC leverages human corrections to construct and refine sets of desired actions, and optimizes a policy to place probability mass over these sets rather than over a single action target. This formulation naturally accommodates both absolute and relative corrections and can represent complex multi-modal behaviors. Extensive simulation and real-robot experiments show that the proposed approach leads to effective policy learning across diverse settings: CLIC remains competitive with the state of the art under accurate data while being substantially more robust under noisy, relative, and partial feedback. Our implementation is publicly available at https://clic-webpage.github.io/.
