NFPO: 正規化フローによるロボットポリシー学習の安定化ポリシー最適化
NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learning
ロボットの強化学習で、多峰性分布を表現できる正規化フローをポリシーとして用いる際の訓練不安定性を分析し、簡単な手法で安定化させて性能を向上させた。
詳しい要約
1. どんなもの？
2. 先行研究と比べてどこがすごい？
3. 技術・手法の肝は？
4. どうやって有効だと検証した？
5. 議論はある？
6. 次に読むべき論文は？
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Diyuan Shi, Yiqi Tang, Zifeng Zhuang, Donglin Wang
分類: cs.RO
原文アブストラクト
Deep Reinforcement Learning (DRL) has experienced significant advancements in recent years and has been widely used in many fields. In DRL-based robotic policy learning, however, current de facto policy parameterization is still multivariate Gaussian (with diagonal covariance matrix), which lacks the ability to model multi-modal distribution. In this work, we explore the adoption of a modern network architecture, i.e. Normalizing Flow (NF) as the policy parameterization for its ability of multi-modal modeling, closed form of log probability and low computation and memory overhead. However, naively training NF in online Reinforcement Learning (RL) usually leads to training instability. We provide a detailed analysis for this phenomenon and successfully address it via simple but effective technique. With extensive experiments in multiple simulation environments, we show our method, NFPO could obtain robust and strong performance in widely used robotic learning tasks and successfully transfer into real-world robots.
