エージェントは見知らぬエージェントとリレーできるか?強化学習の分布外軌道への汎化
Can Agents Run Relay Race with Strangers? Generalization of RL to Out-of-Distribution Trajectories
強化学習エージェントが他のエージェントの軌道途中から制御を引き継ぐ「リレー汎化」を評価し、自己軌道拡張(STA)で改善する手法を提案。
著者: Li-Cheng Lan, Huan Zhang, Cho-Jui Hsieh
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
In this paper, we define, evaluate, and improve the ``relay-generalization'' performance of reinforcement learning (RL) agents on the out-of-distribution ``controllable'' states. Ideally, an RL agent that generally masters a task should reach its goal starting from any controllable state of the environment instead of memorizing a small set of trajectories. For example, a self-driving system should be able to take over the control from humans in the middle of driving and must continue to drive the car safely. To practically evaluate this type of generalization, we start the test agent from the middle of other independently well-trained \emph{stranger} agents' trajectories. With extensive experimental evaluation, we show the prevalence of \emph{generalization failure} on controllable states from stranger agents. For example, in the Humanoid environment, we observed that a well-trained Proximal Policy Optimization (PPO) agent, with only 3.9\% failure rate during regular testing, failed on 81.6\% of the states generated by well-trained stranger PPO agents. To improve "relay generalization," we propose a novel method called Self-Trajectory Augmentation (STA), which will reset the environment to the agent's old states according to the Q function during training. After applying STA to the Soft Actor Critic's (SAC) training procedure, we reduced the failure rate of SAC under relay-evaluation by more than three times in most settings without impacting agent performance and increasing the needed number of environment interactions. Our code is available at https://github.com/lan-lc/STA.
