敵対的オプション認識型階層模倣学習
Adversarial Option-Aware Hierarchical Imitation Learning
長期の未注釈デモンストレーションからスキルを学ぶため、オプションで階層をモデル化し生成的敵対的最適化で方策を訓練するOption-GAILを提案。EM型アルゴリズムで専門家とエージェントのオプション占有測度を最小化し、収束性を理論的に証明した。
著者: Mingxuan Jing, Wenbing Huang, Fuchun Sun, Xiaojian Ma, Tao Kong, Chuang Gan, Lei Li
分類: cs.LG, cs.RO
原文アブストラクト
It has been a challenge to learning skills for an agent from long-horizon unannotated demonstrations. Existing approaches like Hierarchical Imitation Learning(HIL) are prone to compounding errors or suboptimal solutions. In this paper, we propose Option-GAIL, a novel method to learn skills at long horizon. The key idea of Option-GAIL is modeling the task hierarchy by options and train the policy via generative adversarial optimization. In particular, we propose an Expectation-Maximization(EM)-style algorithm: an E-step that samples the options of expert conditioned on the current learned policy, and an M-step that updates the low- and high-level policies of agent simultaneously to minimize the newly proposed option-occupancy measurement between the expert and the agent. We theoretically prove the convergence of the proposed algorithm. Experiments show that Option-GAIL outperforms other counterparts consistently across a variety of tasks.
