分割統治による報酬設計の簡素化
Simplifying Reward Design through Divide-and-Conquer
環境ごとに個別に報酬を設計し、それらを観測として共通報酬を推定する分割統治型の報酬設計手法を提案し、ユーザ実験で従来の一括設計より高速・高品質であることを示した。
著者: Ellis Ratner, Dylan Hadfield-Menell, Anca D. Dragan
分類: cs.RO, cs.AI, cs.LG
原文アブストラクト
Designing a good reward function is essential to robot planning and reinforcement learning, but it can also be challenging and frustrating. The reward needs to work across multiple different environments, and that often requires many iterations of tuning. We introduce a novel divide-and-conquer approach that enables the designer to specify a reward separately for each environment. By treating these separate reward functions as observations about the underlying true reward, we derive an approach to infer a common reward across all environments. We conduct user studies in an abstract grid world domain and in a motion planning domain for a 7-DOF manipulator that measure user effort and solution quality. We show that our method is faster, easier to use, and produces a higher quality solution than the typical method of designing a reward jointly across all environments. We additionally conduct a series of experiments that measure the sensitivity of these results to different properties of the reward design task, such as the number of environments, the number of feasible solutions per environment, and the fraction of the total features that vary within each environment. We find that independent reward design outperforms the standard, joint, reward design process but works best when the design problem can be divided into simpler subproblems.
関連論文
- 進捗関係からロボットの報酬を学習するEvent2Reward強化学習/報酬設計
- ROOT: ユーザー指定の身体動作のための報酬発見フレームワーク強化学習/報酬設計
- 報酬設計エージェント:強化学習のための報酬設計強化学習/報酬設計
- 強化学習のための段階遷移型高密度報酬モデリング強化学習/報酬設計
- 生成的エピソードガイダンスによる二重粒度コントラスト報酬で効率的な身体性RL強化学習/報酬設計
- グラフ思考による報酬進化:強化学習のための二段階言語モデルフレームワーク強化学習/報酬設計
