ニューラルリアプノフ関数と最適制御
Neural Lyapunov and Optimal Control
強化学習の収束の遅さを改善するため、ハミルトン・ヤコビ・ベルマン方程式と一次勾配を用いて時変価値関数と方策を学習する最適制御ベースの手法を提案し、SACやPPOより優れた性能を示した。
著者: Daniel Layeghi, Steve Tonneau, Michael Mistry
分類: cs.RO
原文アブストラクト
Despite impressive results, reinforcement learning (RL) suffers from slow convergence and requires a large variety of tuning strategies. In this paper, we investigate the ability of RL algorithms on simple continuous control tasks. We show that without reward and environment tuning, RL suffers from poor convergence. In turn, we introduce an optimal control (OC) theoretic learning-based method that can solve the same problems robustly with simple parsimonious costs. We use the Hamilton-Jacobi-Bellman (HJB) and first-order gradients to learn optimal time-varying value functions and therefore, policies. We show the relaxation of our objective results in time-varying Lyapunov functions, further verifying our approach by providing guarantees over a compact set of initial conditions. We compare our method to Soft Actor Critic (SAC) and Proximal Policy Optimisation (PPO). In this comparison, we solve all tasks, we never underperform in task cost and we show that at the point of our convergence, we outperform SAC and PPO in the best case by 4 and 2 orders of magnitude.
