マップレスナビゲーションにおける安全な深層強化学習のためのオンライン安全特性の収集と洗練
Online Safety Property Collection and Refinement for Safe Deep Reinforcement Learning in Mapless Navigation
訓練時にコスト信号から安全特性を自動生成・統合するCROPフレームワークを提案し、マップレスナビゲーションで従来の安全DRLより高い報酬と少ない違反を実現した。
著者: Luca Marzari, Enrico Marchesini, Alessandro Farinelli
分類: cs.AI, cs.RO
原文アブストラクト
Safety is essential for deploying Deep Reinforcement Learning (DRL) algorithms in real-world scenarios. Recently, verification approaches have been proposed to allow quantifying the number of violations of a DRL policy over input-output relationships, called properties. However, such properties are hard-coded and require task-level knowledge, making their application intractable in challenging safety-critical tasks. To this end, we introduce the Collection and Refinement of Online Properties (CROP) framework to design properties at training time. CROP employs a cost signal to identify unsafe interactions and use them to shape safety properties. Hence, we propose a refinement strategy to combine properties that model similar unsafe interactions. Our evaluation compares the benefits of computing the number of violations using standard hard-coded properties and the ones generated with CROP. We evaluate our approach in several robotic mapless navigation tasks and demonstrate that the violation metric computed with CROP allows higher returns and lower violations over previous Safe DRL approaches.
