LEMUR: 嗜好フィードバックを用いた多目的強化学習によるアライメント学習
LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback
複数の競合する目的を持つタスクにおいて、複数の人間の嗜好フィードバックから報酬モデルと方策を同時に学習する新しいフレームワークを提案し、ベンチマークタスクで優れた性能を示した。
詳しい要約
1. どんなもの？
2. 先行研究と比べてどこがすごい？
3. 技術・手法の肝は？
4. どうやって有効だと検証した？
5. 議論はある？
6. 次に読むべき論文は？
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Manith Adikari, Bei Peng, Samuele Vinanzi, Angelo Cangelosi
分類: cs.AI, cs.RO
原文アブストラクト
Reinforcement Learning (RL) systems are typically trained using a single, well-specified scalar reward function. However, real-world decision-making tasks often involve multiple, competing objectives, such as performance versus efficiency, where ground-truth reward functions are difficult to specify or inaccessible. While Multi-Objective RL (MORL) addresses such trade-offs by modeling rewards as vectors, existing approaches typically assume access to a well-specified reward function for each objective, inheriting the same challenges faced by single-objective RL. Meanwhile, Preference-based RL (PbRL) has shown great potential in solving complex tasks without access to a pre-defined reward function through reward learning from human feedback, yet has largely been studied in single-objective settings. In this work, we bridge this gap with LEMUR: Learning to Align with Multi-Objective Reinforcement Learning with Preference feedback, a novel framework where an agent interactively learns from the preferences of multiple humans to learn optimal multi-objective policies. Our approach jointly learns policies and multiple objective-specific reward models from human feedback, enabling agents to effectively balance competing objectives during learning. We evaluate LEMUR on a variety of benchmark multi-objective tasks, and empirical results demonstrate its superior performance over baseline methods. Our method presents a promising direction for solving multi-objective decision-making tasks without pre-defined reward functions.
関連論文
- ルーティング型アンサンブルと軌跡帰属による多目的強化学習多目的強化学習
- 多目的方策最適化のための適応的スムーズ・チェビシェフ注意機構多目的強化学習
- 後知恵選好リプレイによる選好条件付き多目的強化学習の改善多目的強化学習
- 多目的連続ロボット制御のためのパレート集合学習多目的強化学習
- 多目的方策最適化の分布的アプローチ多目的強化学習
- 報酬関数の補間による多目的強化学習の最適価値関数予測多目的強化学習
