専門家を超えよ:並行方策最適化を伴うステレオ逆強化学習
Learn to Exceed: Stereo Inverse Reinforcement Learning with Concurrent Policy Optimization
専門家のデモンストレーションから報酬と方策を同時に学習し、専門家を上回る制御方策を獲得する手法を提案。OpenAI環境と屋内ドローン飛行で有効性を検証した。
著者: Feng Tao, Yongcan Cao
分類: cs.LG, cs.RO, stat.ML
原文アブストラクト
In this paper, we study the problem of obtaining a control policy that can mimic and then outperform expert demonstrations in Markov decision processes where the reward function is unknown to the learning agent. One main relevant approach is the inverse reinforcement learning (IRL), which mainly focuses on inferring a reward function from expert demonstrations. The obtained control policy by IRL and the associated algorithms, however, can hardly outperform expert demonstrations. To overcome this limitation, we propose a novel method that enables the learning agent to outperform the demonstrator via a new concurrent reward and action policy learning approach. In particular, we first propose a new stereo utility definition that aims to address the bias in the interpretation of expert demonstrations. We then propose a loss function for the learning agent to learn reward and action policies concurrently such that the learning agent can outperform expert demonstrations. The performance of the proposed method is first demonstrated in OpenAI environments. Further efforts are conducted to experimentally validate the proposed method via an indoor drone flight scenario.
