慎重なベイズ最適化による効率的でスケーラブルな方策探索
Cautious Bayesian Optimization for Efficient and Scalable Policy Search
ベイズ最適化の方策探索空間を予測不確実性の劣位集合に制限することで、高次元空間でもスケールし、システム破損リスクを抑える手法を提案し、モータースキルやsim2real課題で有効性を示した。
著者: Lukas P. Fröhlich, Melanie N. Zeilinger, Edgar D. Klenske
分類: cs.RO
原文アブストラクト
Sample efficiency is one of the key factors when applying policy search to real-world problems. In recent years, Bayesian Optimization (BO) has become prominent in the field of robotics due to its sample efficiency and little prior knowledge needed. However, one drawback of BO is its poor performance on high-dimensional search spaces as it focuses on global search. In the policy search setting, local optimization is typically sufficient as initial policies are often available, e.g., via meta-learning, kinesthetic demonstrations or sim-to-real approaches. In this paper, we propose to constrain the policy search space to a sublevel-set of the Bayesian surrogate model's predictive uncertainty. This simple yet effective way of constraining the policy update enables BO to scale to high-dimensional spaces (>100) as well as reduces the risk of damaging the system. We demonstrate the effectiveness of our approach on a wide range of problems, including a motor skills task, adapting deep RL agents to new reward signals and a sim-to-real task for an inverted pendulum system.
