状態拡張による制約付き強化学習:報酬学習の限界を克服する
State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewards
報酬の線形結合では最適方策が得られない制約付き強化学習問題に対し、状態にラグランジュ乗数を追加して双対ダイナミクスとして解く手法を提案し、最適方策のサンプリングを保証する。
著者: Miguel Calvo-Fullana, Santiago Paternain, Luiz F. O. Chamon, Alejandro Ribeiro
分類: cs.LG, cs.RO, math.OC
原文アブストラクト
A common formulation of constrained reinforcement learning involves multiple rewards that must individually accumulate to given thresholds. In this class of problems, we show a simple example in which the desired optimal policy cannot be induced by any weighted linear combination of rewards. Hence, there exist constrained reinforcement learning problems for which neither regularized nor classical primal-dual methods yield optimal policies. This work addresses this shortcoming by augmenting the state with Lagrange multipliers and reinterpreting primal-dual methods as the portion of the dynamics that drives the multipliers evolution. This approach provides a systematic state augmentation procedure that is guaranteed to solve reinforcement learning problems with constraints. Thus, as we illustrate by an example, while previous methods can fail at finding optimal policies, running the dual dynamics while executing the augmented policy yields an algorithm that provably samples actions from the optimal policy.
関連論文
- 制約付き強化学習のための確率的決定ホライズン制約付き強化学習
- 敵対的制約付き方策最適化:バジェット適応による制約付き強化学習の改善制約付き強化学習
- 適応的ポリシー切替による強化学習における時相論理制約の確率的充足制約付き強化学習
- ペナルティ指標ネットワークを用いた外点ペナルティ方策最適化制約付き強化学習
- 平滑化対数バリア関数を用いた制約付き強化学習制約付き強化学習
- 到達可能性に基づく制約付き強化学習制約付き強化学習
