不確実性に基づく平滑方策正則化:少数のデモンストレーションを用いた強化学習
Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations
デモンストレーションと自身の方策のどちらを模倣すべきかをQ値の不確実性で判断し、連続的な正則化重みで学習を加速する手法SPReDを提案。8つのロボティクスタスクで最大14倍の性能向上を達成。
詳しい要約
1. どんなもの?
2. 先行研究と比べてどこがすごい?
3. 技術・手法の肝は?
4. どうやって有効だと検証した?
5. 議論はある?
6. 次に読むべき論文は?
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Yujie Zhu, Charles A. Hepburn, Matthew Thorpe, Giovanni Montana
分類: cs.LG, cs.AI, cs.RO, stat.ML
原文アブストラクト
In reinforcement learning with sparse rewards, demonstrations can accelerate learning, but determining when to imitate them remains challenging. We propose Smooth Policy Regularisation from Demonstrations (SPReD), a framework that addresses the fundamental question: when should an agent imitate a demonstration versus follow its own policy? SPReD uses ensemble methods to explicitly model Q-value distributions for both demonstration and policy actions, quantifying uncertainty for comparisons. We develop two complementary uncertainty-aware methods: a probabilistic approach estimating the likelihood of demonstration superiority, and an advantage-based approach scaling imitation by statistical significance. Unlike prevailing methods (e.g. Q-filter) that make binary imitation decisions, SPReD applies continuous, uncertainty-proportional regularisation weights, reducing gradient variance during training. Despite its computational simplicity, SPReD achieves remarkable gains in experiments across eight robotics tasks, outperforming existing approaches by up to a factor of 14 in complex tasks while maintaining robustness to demonstration quality and quantity. Our code is available at https://github.com/YujieZhu7/SPReD.
