FlowRL: 拡散ポリシーを用いた強化学習の分類法とモジュール型フレームワーク
FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policies
拡散モデルやフローモデルをポリシーとして用いる強化学習アルゴリズムを体系的に分類し、JAXベースのモジュール型コードベースと標準ベンチマークを提供した論文。
詳しい要約
1. どんなもの?
2. 先行研究と比べてどこがすごい?
3. 技術・手法の肝は?
4. どうやって有効だと検証した?
5. 議論はある?
6. 次に読むべき論文は?
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Chenxiao Gao, Edward Chen, Tianyi Chen, Bo Dai
分類: cs.LG
原文アブストラクト
Thanks to their remarkable flexibility, diffusion models and flow models have emerged as promising candidates for policy representation. However, efficient reinforcement learning (RL) upon these policies remains a challenge due to the lack of explicit log-probabilities for vanilla policy gradient estimators. While numerous attempts have been proposed to address this, the field lacks a unified perspective to reconcile these seemingly disparate methods, thus hampering ongoing development. In this paper, we bridge this gap by introducing a comprehensive taxonomy for RL algorithms with diffusion/flow policies. To support reproducibility and agile prototyping, we introduce a modular, JAX-based open-source codebase that leverages JIT-compilation for high-throughput training. Finally, we provide systematic and standardized benchmarks across Gym-Locomotion, DeepMind Control Suite, and IsaacLab, offering a rigorous side-by-side comparison of diffusion-based methods and guidance for practitioners to choose proper algorithms based on the application. Our work establishes a clear foundation for understanding and algorithm design, a high-efficiency toolkit for future research in the field, and an algorithmic guideline for practitioners in generative models and robotics. Our code is available at https://github.com/typoverflow/flow-rl.
関連論文
- QPILOTS: フローポリシーのための効率的なテスト時Qステアリング強化学習/拡散ポリシー
