観測シフトを超えた強化学習の汎化
Generalization Across Observation Shifts in Reinforcement Learning
双模倣指標を拡張し、文脈依存の観測シフトに不変な表現空間を学習することで、テスト時の観測変化や未見環境への汎化を実現する手法を提案。
著者: Anuj Mahajan, Amy Zhang
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
Learning policies which are robust to changes in the environment are critical for real world deployment of Reinforcement Learning agents. They are also necessary for achieving good generalization across environment shifts. We focus on bisimulation metrics, which provide a powerful means for abstracting task relevant components of the observation and learning a succinct representation space for training the agent using reinforcement learning. In this work, we extend the bisimulation framework to also account for context dependent observation shifts. Specifically, we focus on the simulator based learning setting and use alternate observations to learn a representation space which is invariant to observation shifts using a novel bisimulation based objective. This allows us to deploy the agent to varying observation settings during test time and generalize to unseen scenarios. We further provide novel theoretical bounds for simulator fidelity and performance transfer guarantees for using a learnt policy to unseen shifts. Empirical analysis on the high-dimensional image based control domains demonstrates the efficacy of our method.
