生成モデルと強化学習を組み合わせたデータ効率の良い視覚運動方策学習
Data-efficient visuomotor policy training using reinforcement learning and generative models
行動潜在変数を用いて強化学習と生成モデルを組み合わせ、ロボットの視覚運動方策をデータ効率良く学習する枠組みを提案し、潜在表現の品質評価指標も導入した。
著者: Ali Ghadirzadeh, Petra Poklukar, Ville Kyrki, Danica Kragic, Mårten Björkman
分類: cs.RO, cs.LG
原文アブストラクト
We present a data-efficient framework for solving visuomotor sequential decision-making problems which exploits the combination of reinforcement learning (RL) and latent variable generative models. Our framework trains deep visuomotor policies by introducing an action latent variable such that the feed-forward policy search can be divided into three parts: (i) training a sub-policy that outputs a distribution over the action latent variable given a state of the system, (ii) unsupervised training of a generative model that outputs a sequence of motor actions conditioned on the latent action variable, and (iii) supervised training of the deep visuomotor policy in an end-to-end fashion. Our approach enables safe exploration and alleviates the data-inefficiency problem as it exploits prior knowledge about valid sequences of motor actions. Moreover, we provide a set of measures for evaluation of generative models such that we are able to predict the performance of the RL policy training prior to the actual training on a physical robot. We define two novel measures of disentanglement and local linearity for assessing the quality of latent representations, and complement them with existing measures for assessment of the learned distribution. We experimentally determine the characteristics of different generative models that have the most influence on performance of the final policy training on a robotic picking task.
