フィードバック操作正則化:模倣学習におけるオフラインエージェントのアライメントを可能にする
Feedback Manipulation Regularization: Enabling Offline Agent Alignment for Imitation Learning
評価フィードバックを補正信号として利用し、模倣学習ポリシーのアライメントを改善するアルゴリズム非依存の手法FMRを提案し、Safety Gymnasium環境で最大98%の不整合削減を実証した。
詳しい要約
1. どんなもの?
2. 先行研究と比べてどこがすごい?
3. 技術・手法の肝は?
4. どうやって有効だと検証した?
5. 議論はある?
6. 次に読むべき論文は?
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Benjamin Poole, Minwoo Lee
分類: cs.AI, cs.HC, cs.LG
原文アブストラクト
Reinforcement learning (RL) research has increasingly shifted focus towards alignment, ensuring agents learn behaviors adhering to human values. While human demonstrations and feedback have proven crucial for alignment, existing approaches predominantly combine these signals using multi-stage pipelines designed for the contextual bandit framing of language generation. Yet little work explores how these complementary inputs can serve as a richer, interconnected signal for single-stage offline training in fully sequential decision-making environments. We propose Feedback Manipulation Regularization (FMR), an algorithm-agnostic method that harnesses evaluative feedback as a corrective signal to improve the alignment of imitation learning policies. We adapt Safety Gymnasium environments to be a principled testbed for alignment evaluation, demonstrating improved aptitude and up to a 98\% reduction in misalignment across a range of imitation learning algorithms. FMR remains robust in limited data regimes, even when learning from scarce aligned and uninformative noisy demonstrations.
