モデルアンサンブルによるロバストなニューラルネットワーク方策の学習
EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
複数のシミュレーションモデルをアンサンブルし敵対的訓練を行うことで、未知の環境変化にも頑健に汎化する強化学習方策を学習するEPOptを提案した。
著者: Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, Sergey Levine
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
Sample complexity and safety are major challenges when learning policies with reinforcement learning for real-world tasks, especially when the policies are represented using rich function approximators like deep neural networks. Model-based methods where the real-world target domain is approximated using a simulated source domain provide an avenue to tackle the above challenges by augmenting real data with simulated data. However, discrepancies between the simulated source domain and the target domain pose a challenge for simulated training. We introduce the EPOpt algorithm, which uses an ensemble of simulated source domains and a form of adversarial training to learn policies that are robust and generalize to a broad range of possible target domains, including unmodeled effects. Further, the probability distribution over source domains in the ensemble can be adapted using data from target domain and approximate Bayesian methods, to progressively make it a better approximation. Thus, learning on a model ensemble, along with source domain adaptation, provides the benefit of both robustness and learning/adaptation.
