動きながら考える：並行制御による深層強化学習
Thinking While Moving: Deep Reinforcement Learning with Concurrent Control
行動の実行中に次の行動を決定する並行制御問題に対し、連続時間ベルマン方程式を遅延考慮で離散化する強化学習手法を提案し、ロボット把持タスクで有効性を示した。
著者: Ted Xiao, Eric Jang, Dmitry Kalashnikov, Sergey Levine, Julian Ibarz, Karol Hausman, Alexander Herzog
分類: cs.LG, cs.AI, cs.RO, stat.ML
原文アブストラクト
We study reinforcement learning in settings where sampling an action from the policy must be done concurrently with the time evolution of the controlled system, such as when a robot must decide on the next action while still performing the previous action. Much like a person or an animal, the robot must think and move at the same time, deciding on its next action before the previous one has completed. In order to develop an algorithmic framework for such concurrent control problems, we start with a continuous-time formulation of the Bellman equations, and then discretize them in a way that is aware of system delays. We instantiate this new class of approximate dynamic programming methods via a simple architectural extension to existing value-based deep reinforcement learning algorithms. We evaluate our methods on simulated benchmark tasks and a large-scale robotic grasping task where the robot must "think while moving".
