計算効率を両立する任意の確率方策に向けた制限付き正規化フローの設計
Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiency
正規化フローを適切に制限して解析的平均を可能にし、二峰性student-t分布をベースに用いることで表現力と計算効率を両立した確率方策Bit-RNFを提案し、強化学習ベンチマークと実機ロボット実験で有効性を示した。
著者: Taisuke Kobayashi, Takumi Aotani
分類: cs.RO, cs.LG
原文アブストラクト
This paper proposes a new design method for a stochastic control policy using a normalizing flow (NF). In reinforcement learning (RL), the policy is usually modeled as a distribution model with trainable parameters. When this parameterization has less expressiveness, it would fail to acquiring the optimal policy. A mixture model has capability of a universal approximation, but it with too much redundancy increases the computational cost, which can become a bottleneck when considering the use of real-time robot control. As another approach, NF, which is with additional parameters for invertible transformation from a simple stochastic model as a base, is expected to exert high expressiveness and lower computational cost. However, NF cannot compute its mean analytically due to complexity of the invertible transformation, and it lacks reliability because it retains stochastic behaviors after deployment for robot controller. This paper therefore designs a restricted NF (RNF) that achieves an analytic mean by appropriately restricting the invertible transformation. In addition, the expressiveness impaired by this restriction is regained using bimodal student-t distribution as its base, so-called Bit-RNF. In RL benchmarks, Bit-RNF policy outperformed the previous models. Finally, a real robot experiment demonstrated the applicability of Bit-RNF policy to real world. The attached video is uploaded on youtube: https://youtu.be/R_GJVZDW9bk
