射影に基づく制約付き方策最適化
Projection-Based Constrained Policy Optimization
報酬改善の更新と制約違反を射影で補正する2段階の反復手法PCPOを提案し、理論的な性能保証と収束性を示した上で、複数の制御タスクで制約違反を大幅に削減しつつ報酬を向上させた。
著者: Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, Peter J. Ramadge
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
We consider the problem of learning control policies that optimize a reward function while satisfying constraints due to considerations of safety, fairness, or other costs. We propose a new algorithm, Projection-Based Constrained Policy Optimization (PCPO). This is an iterative method for optimizing policies in a two-step process: the first step performs a local reward improvement update, while the second step reconciles any constraint violation by projecting the policy back onto the constraint set. We theoretically analyze PCPO and provide a lower bound on reward improvement, and an upper bound on constraint violation, for each policy update. We further characterize the convergence of PCPO based on two different metrics: $\normltwo$ norm and Kullback-Leibler divergence. Our empirical results over several control tasks demonstrate that PCPO achieves superior performance, averaging more than 3.5 times less constraint violation and around 15\% higher reward compared to state-of-the-art methods.
関連論文
- 制約付き強化学習のための確率的決定ホライズン制約付き強化学習
- 敵対的制約付き方策最適化:バジェット適応による制約付き強化学習の改善制約付き強化学習
- 適応的ポリシー切替による強化学習における時相論理制約の確率的充足制約付き強化学習
- ペナルティ指標ネットワークを用いた外点ペナルティ方策最適化制約付き強化学習
- 平滑化対数バリア関数を用いた制約付き強化学習制約付き強化学習
- 到達可能性に基づく制約付き強化学習制約付き強化学習
