学習報酬を用いた大規模行動モデルのコヒーレントなオフポリシー改善
Coherent Off-Policy Improvement of Large Behavior Models with Learned Rewards
大規模な事前学習済みロボット操作ポリシーを、逆強化学習で学習した稠密報酬を用いて微調整し、スパース報酬のRLベースラインよりも高い成功率を達成した。
詳しい要約
1. どんなもの？
2. 先行研究と比べてどこがすごい？
3. 技術・手法の肝は？
4. どうやって有効だと検証した？
5. 議論はある？
6. 次に読むべき論文は？
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Christian Scherer, Joe Watson, Theo Gruner, Daniel Palenicek, Ingmar Posner, Jan Peters
分類: cs.LG
原文アブストラクト
Distilling expert demonstration data into large generative models using behavioral cloning is a scalable approach to learning capable policies for robotic control, particularly for dexterous manipulation. Reinforcement learning (RL) can be used as a means to finetune these policies further using additional experience. An open question is whether RL is more sample-efficient than collecting more human demonstrations. Prior work has finetuned large pretrained policies in a scalable fashion by applying RL to a smaller residual policy that corrects the pretrained model. However, for the typical sparse reward tasks, RL algorithms can struggle to optimize the behavior in a sample-efficient manner. We explore inverse reinforcement learning, where a dense reward function is learned from expert demonstrations, potentially reducing the challenge of RL finetuning. We specifically consider coherent imitation learning, an IRL method that facilitates improvement of the BC policy through using a specific reward formulation with theoretical guarantees. We show that our IRL method maintains or improves the performance of pi-0.5 on all six sparse manipulation tasks and achieves a $\geq 90\%$ success rate on five out of six complex manipulation tasks, outperforming RL-based baselines using sparse rewards. By ensuring our initial pretrained finetuning policy is optimal for our initial reward and critic, our method circumvents the initial drop commonly seen in RL finetuning and enables faster improvement.
関連論文
- 拡散ベースのポリシーにおける隠れた報酬の回復模倣学習/逆強化学習
- MEReQ: 人間の介入からサンプル効率よく方針を整合させる最大エントロピー残差Q逆強化学習模倣学習/逆強化学習
- 変分逆強化学習による敵対的模倣学習模倣学習/逆強化学習
