構造的最大化可能なQ関数によるアクターフリー連続制御
Actor-Free Continuous Control via Structurally Maximizable Q-Functions
連続行動空間での制御において、アクターを別途学習せずにQ関数の構造的最大化を利用する純粋な価値ベース強化学習フレームワークを提案し、標準的なシミュレーションタスクで従来手法と同等以上の性能とサンプル効率を達成した。
詳しい要約
1. どんなもの?
2. 先行研究と比べてどこがすごい?
3. 技術・手法の肝は?
4. どうやって有効だと検証した?
5. 議論はある?
6. 次に読むべき論文は?
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Yigit Korkmaz, Urvi Bhuwania, Ayush Jain, Erdem Bıyık
分類: cs.LG, cs.AI, cs.RO, stat.ML
原文アブストラクト
Value-based algorithms are a cornerstone of off-policy reinforcement learning due to their simplicity and training stability. However, their use has traditionally been restricted to discrete action spaces, as they rely on estimating Q-values for individual state-action pairs. In continuous action spaces, evaluating the Q-value over the entire action space becomes computationally infeasible. To address this, actor-critic methods are typically employed, where a critic is trained on off-policy data to estimate Q-values, and an actor is trained to maximize the critic's output. Despite their popularity, these methods often suffer from instability during training. In this work, we propose a purely value-based framework for continuous control that revisits structural maximization of Q-functions, introducing a set of key architectural and algorithmic choices to enable efficient and stable learning. We evaluate the proposed actor-free Q-learning approach on a range of standard simulation tasks, demonstrating performance and sample efficiency on par with state-of-the-art baselines, without the cost of learning a separate actor. Particularly, in environments with constrained action spaces, where the value functions are typically non-smooth, our method with structural maximization outperforms traditional actor-critic methods with gradient-based maximization. We have released our code at https://github.com/USC-Lira/Q3C.
関連論文
- 行動前に選択せよ:連続制御のための空間的に分離された行動反復連続制御/強化学習
