フロー補完による単一ステップ生成方策:効率的で表現力豊かな方策学習
Flow-Based Single-Step Completion for Efficient and Expressive Policy Learning
拡張フローマッチング目的で中間フローから直接補完ベクトルを予測し、1回の推論で多様な行動を生成できる生成方策SSCPを提案。オフライン・オンラインRLで拡散モデルより高速かつ高精度を実現。
詳しい要約
1. どんなもの?
2. 先行研究と比べてどこがすごい?
3. 技術・手法の肝は?
4. どうやって有効だと検証した?
5. 議論はある?
6. 次に読むべき論文は?
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Prajwal Koirala, Cody Fleming
分類: cs.LG, cs.RO
原文アブストラクト
Generative models such as diffusion and flow-matching offer expressive policies for offline reinforcement learning (RL) by capturing rich, multimodal action distributions, but their iterative sampling introduces high inference costs and training instability due to gradient propagation across sampling steps. We propose the Single-Step Completion Policy (SSCP), a generative policy trained with an augmented flow-matching objective to predict direct completion vectors from intermediate flow samples, enabling accurate, one-shot action generation. In an off-policy actor-critic framework, SSCP combines the expressiveness of generative models with the training and inference efficiency of unimodal policies, without requiring long backpropagation chains. Our method scales effectively to offline, offline-to-online, and online RL settings, offering substantial gains in speed and adaptability over diffusion-based baselines. We further extend SSCP to goal-conditioned RL, enabling flat policies to exploit subgoal structures without explicit hierarchical inference. SSCP achieves strong results across standard offline RL and behavior cloning benchmarks, positioning it as a versatile, expressive, and efficient framework for deep RL and sequential decision-making. The code is available at https://github.com/PrajwalKoirala/SSCP-Single-Step-Completion-Policy.
