実例からの価値ペナルティ付き補助制御によるデモンストレーション不要の効率的模倣
Efficient Imitation Without Demonstrations via Value-Penalized Auxiliary Control from Examples
完了したタスクの実例のみを用いた強化学習において、簡単な補助タスクの実例と成功レベル以上の価値ペナルティを追加することで探索を大幅に改善し、学習効率を高める手法VPACEを提案した。
著者: Trevor Ablett, Bryan Chan, Jayce Haoran Wang, Jonathan Kelly
分類: cs.RO, cs.AI, cs.LG
原文アブストラクト
Common approaches to providing feedback in reinforcement learning are the use of hand-crafted rewards or full-trajectory expert demonstrations. Alternatively, one can use examples of completed tasks, but such an approach can be extremely sample inefficient. We introduce value-penalized auxiliary control from examples (VPACE), an algorithm that significantly improves exploration in example-based control by adding examples of simple auxiliary tasks and an above-success-level value penalty. Across both simulated and real robotic environments, we show that our approach substantially improves learning efficiency for challenging tasks, while maintaining bounded value estimates. Preliminary results also suggest that VPACE may learn more efficiently than the more common approaches of using full trajectories or true sparse rewards. Project site: https://papers.starslab.ca/vpace/.
