強化学習における足場的リフレクションによる閉鎖空間脱出問題の解法
Scaffolding Reflection in Reinforcement Learning Framework for Confinement Escape Problem
追跡者に囲まれた閉鎖領域から逃走者が脱出する問題に対し、運動プランナを足場として俳優-批評家ネットワークの学習を補助するSR2L手法を提案し、報酬の疎性による収束困難を改善した。
著者: Nishant Mohanty, Suresh Sundaram
分類: cs.RO, cs.LG
原文アブストラクト
In this paper, a novel Scaffolding Reflection in Reinforcement Learning (SR2L) is proposed for solving the confinement escape problem (CEP). In CEP, an evader's objective is to attempt escaping a confinement region patrolled by multiple pursuers. Meanwhile, the pursuers aim to reach and capture the evader. The inverse solution for pursuers to try and capture has been extensively studied in the literature. However, the problem of evaders escaping from the region is still an open issue. The SR2L employs an actor-critic framework to enable the evader to escape the confinement region. A time-varying state representation and reward function have been developed for proper convergence. The formulation uses the sensor information about the observable environment and prior knowledge of the confinement boundary. The conventional Independent Actor-Critic (IAC) method fails to converge due to sparseness in the reward. The effect becomes evident when operating in such a dynamic environment with a large area. In SR2L, along with the developed reward function, we use the scaffolding reflection method to improve the convergence significantly while increasing its efficiency. In SR2L, a motion planner is used as a scaffold for the actor-critic network to observe, compare and learn the action-reward pair. It enables the evader to achieve the required objective while using lesser resources and time. Convergence studies show that SR2L learns faster and converges to higher rewards as compared to IAC. Extensive Monte-Carlo simulations show that a SR2L consistently outperforms conventional IAC and the motion planner itself as the baselines.
