模倣学習と強化学習における潜在交絡因子の共変量シフトについて
On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learning
未観測の交絡因子を含む専門家データからの模倣・強化学習を定式化し、分布シフト下での学習可能性と不可能性を示すとともに、外部報酬がある場合のサンプリング手法を提案して収束を証明した。
著者: Guy Tennenholtz, Assaf Hallak, Gal Dalal, Shie Mannor, Gal Chechik, Uri Shalit
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
We consider the problem of using expert data with unobserved confounders for imitation and reinforcement learning. We begin by defining the problem of learning from confounded expert data in a contextual MDP setup. We analyze the limitations of learning from such data with and without external reward, and propose an adjustment of standard imitation learning algorithms to fit this setup. We then discuss the problem of distribution shift between the expert data and the online environment when the data is only partially observable. We prove possibility and impossibility results for imitation learning under arbitrary distribution shift of the missing covariates. When additional external reward is provided, we propose a sampling procedure that addresses the unknown shift and prove convergence to an optimal solution. Finally, we validate our claims empirically on challenging assistive healthcare and recommender system simulation tasks.
