正規化フローを用いた観測からの模倣学習
IL-flOw: Imitation Learning from Observation using Normalizing Flows
専門家の状態観測のみから、正規化フローで状態遷移の密度を学習し報酬として使うことで、不安定な敵対的手法を避けて方策を学習する手法を提案した論文。
著者: Wei-Di Chang, Juan Camilo Gamboa Higuera, Scott Fujimoto, David Meger, Gregory Dudek
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
We present an algorithm for Inverse Reinforcement Learning (IRL) from expert state observations only. Our approach decouples reward modelling from policy learning, unlike state-of-the-art adversarial methods which require updating the reward model during policy search and are known to be unstable and difficult to optimize. Our method, IL-flOw, recovers the expert policy by modelling state-state transitions, by generating rewards using deep density estimators trained on the demonstration trajectories, avoiding the instability issues of adversarial methods. We demonstrate that using the state transition log-probability density as a reward signal for forward reinforcement learning translates to matching the trajectory distribution of the expert demonstrations, and experimentally show good recovery of the true reward signal as well as state of the art results for imitation from observation on locomotion and robotic continuous control tasks.
