整列とフィルタリング:非同期オン方策強化学習の性能向上
Align and Filter: Improving Performance in Asynchronous On-Policy RL
分散学習と高頻度更新によるポリシーラグ(データ生成と学習の不整合)を軽減するため、全変動に基づくアドバンテージ整列制約付きポリシー最適化手法を提案し、古典的RLタスクとLLM数学推論タスクで有効性を検証した。
詳しい要約
1. どんなもの?
2. 先行研究と比べてどこがすごい?
3. 技術・手法の肝は?
4. どうやって有効だと検証した?
5. 議論はある?
6. 次に読むべき論文は?
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Homayoun Honari, Roger Creus Castanyer, Michael Przystupa, Michael Noukhovitch, Pablo Samuel Castro, Glen Berseth
分類: cs.LG, cs.AI, cs.RO, cs.SY, eess.SY
原文アブストラクト
Distributed training and increasing the gradient update frequency are practical strategies to accelerate learning and improve performance, but both exacerbate a central challenge: \textit{policy lag}, which is the mismatch between the behavior policy generating data and the learning policy being updated. Policy lag can hinder the scaling of on-policy learning algorithms to larger problems. In this paper, we identify the sources of policy lag caused by distributed learning and high update frequency. We use the findings to propose \textit{total Variation-based Advantage aligned Constrained policy Optimization (\methodacronym)} as a practical approach to mitigate policy lag. We empirically validate our method and show that it offers better robustness to policy lag in classic RL tasks and a modern RL for LLM math reasoning task.
