COP-Q: コレスキー順序射影による安全最優先ロボット制御強化学習
COP-Q: Safety-First Reinforcement Learning for Robot Control via Cholesky-Ordered Projection
報酬と安全性のQ値の相関を考慮し、コレスキー分解で安全を優先しつつ報酬の過度な保守性を抑える新しい安全強化学習手法を提案した。ロボットの移動やナビゲーション実験で安全性とサンプル効率の向上を確認した。
詳しい要約
1. どんなもの?
2. 先行研究と比べてどこがすごい?
3. 技術・手法の肝は?
4. どうやって有効だと検証した?
5. 議論はある?
6. 次に読むべき論文は?
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Guopeng Li, Moritz A. Zanger, Matthijs T. J. Spaan, Julian F. P. Kooij
分類: cs.RO, cs.LG
原文アブストラクト
Safe robot control requires maximizing return while satisfying safety constraints. In off-policy safe reinforcement learning, reward and safety Q-values are commonly learned by separate critic ensembles, with uncertainty handled independently for each objective. This objective-wise treatment neglects inter-objective correlation and can lead to overly conservative value estimates, thereby reducing sample efficiency. To address this issue, we propose Cholesky-Ordered Projection Q-learning (COP-Q), a safety-first method that incorporates inter-objective covariance into vector-valued Q-value estimation. COP-Q constructs a generalized confidence bound in the joint Q-value space and uses Cholesky factorization to encode objective priority in a sequential form. This preserves conservatism on safety while adaptively reducing excessive conservatism on the reward objective. The resulting estimate is used in both temporal-difference target computation and actor optimization. COP-Q incurs minimal computational overhead and is readily compatible with most existing deep Q-learning frameworks. Experiments on robot locomotion in Brax and safe navigation in Safety-Gymnasium, covering both hard- and soft-safety settings, demonstrate that COP-Q achieves strong safety performance together with competitive or improved sample efficiency relative to representative baselines.
