不確実性を考慮した到達可能性証明による安全なモデルベース強化学習
Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificate
モデルの不確実性に対処する分布型到達可能性証明（DRC）を提案し、訓練中の制約違反を抑えつつ高いリターンを得る安全なモデルベース強化学習フレームワークを構築した。
著者: Dongjie Yu, Wenjun Zou, Yujie Yang, Haitong Ma, Shengbo Eben Li, Jingliang Duan, Jianyu Chen
分類: cs.RO, cs.LG, cs.SY, eess.SY
原文アブストラクト
Safe reinforcement learning (RL) that solves constraint-satisfactory policies provides a promising way to the broader safety-critical applications of RL in real-world problems such as robotics. Among all safe RL approaches, model-based methods reduce training time violations further due to their high sample efficiency. However, lacking safety robustness against the model uncertainties remains an issue in safe model-based RL, especially in training time safety. In this paper, we propose a distributional reachability certificate (DRC) and its Bellman equation to address model uncertainties and characterize robust persistently safe states. Furthermore, we build a safe RL framework to resolve constraints required by the DRC and its corresponding shield policy. We also devise a line search method to maintain safety and reach higher returns simultaneously while leveraging the shield policy. Comprehensive experiments on classical benchmarks such as constrained tracking and navigation indicate that the proposed algorithm achieves comparable returns with much fewer constraint violations during training.
