非定常環境における空中ロボット制御のための強化学習の可塑性維持:コスト認識フレームワーク
Maintaining Plasticity in Reinforcement Learning: A Cost-Aware Framework for Aerial Robot Control in Non-stationary Environments
強化学習の長期学習で生じる可塑性喪失を防ぐため、報酬と損失のバランスを取る回顧的コスト機構(RECOM)を提案し、風擾乱下でのホバリング制御でポリシー崩壊を回避した。
著者: Ali Tahir Karasahin, Ziniu Wu, Basaran Bahadir Kocer
分類: cs.RO
原文アブストラクト
Reinforcement learning (RL) has demonstrated the ability to maintain the plasticity of the policy throughout short-term training in aerial robot control. However, these policies have been shown to loss of plasticity when extended to long-term learning in non-stationary environments. For example, the standard proximal policy optimization (PPO) policy is observed to collapse in long-term training settings and lead to significant control performance degradation. To address this problem, this work proposes a cost-aware framework that uses a retrospective cost mechanism (RECOM) to balance rewards and losses in RL training with a non-stationary environment. Using a cost gradient relation between rewards and losses, our framework dynamically updates the learning rate to actively train the control policy in a disturbed wind environment. Our experimental results show that our framework learned a policy for the hovering task without policy collapse in variable wind conditions and has a successful result of 11.29% less dormant units than L2 regularization with PPO.
