潜在拡散モデルによる行動多様性を持つ方策の生成
Generating Behaviorally Diverse Policies with Latent Diffusion Models
品質多様性強化学習で得た多数の方策を拡散モデルで蒸留し、13倍の圧縮率で性能とカバレッジを保ちつつ、言語による行動選択も可能にした。
著者: Shashank Hegde, Sumeet Batra, K. R. Zentner, Gaurav S. Sukhatme
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
Recent progress in Quality Diversity Reinforcement Learning (QD-RL) has enabled learning a collection of behaviorally diverse, high performing policies. However, these methods typically involve storing thousands of policies, which results in high space-complexity and poor scaling to additional behaviors. Condensing the archive into a single model while retaining the performance and coverage of the original collection of policies has proved challenging. In this work, we propose using diffusion models to distill the archive into a single generative model over policy parameters. We show that our method achieves a compression ratio of 13x while recovering 98% of the original rewards and 89% of the original coverage. Further, the conditioning mechanism of diffusion models allows for flexibly selecting and sequencing behaviors, including using language. Project website: https://sites.google.com/view/policydiffusion/home
