物理的微分:物理的な順伝播による方策勾配の計算
Physical Derivatives: Computing policy gradients by physical forward-propagation
遷移モデルの代わりにパラメータ摂動に対する軌道の感度を学習し、実機ロボットで方策勾配を物理的に順伝播して推定する手法を提案・検証した。
著者: Arash Mehrjou, Ashkan Soleymani, Stefan Bauer, Bernhard Schölkopf
分類: cs.RO, math.DS, stat.ML
原文アブストラクト
Model-free and model-based reinforcement learning are two ends of a spectrum. Learning a good policy without a dynamic model can be prohibitively expensive. Learning the dynamic model of a system can reduce the cost of learning the policy, but it can also introduce bias if it is not accurate. We propose a middle ground where instead of the transition model, the sensitivity of the trajectories with respect to the perturbation of the parameters is learned. This allows us to predict the local behavior of the physical system around a set of nominal policies without knowing the actual model. We assay our method on a custom-built physical robot in extensive experiments and show the feasibility of the approach in practice. We investigate potential challenges when applying our method to physical systems and propose solutions to each of them.
