拡散ポリシーによるオフライン強化学習の分布外汎化
Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning
オフライン強化学習において、拡散ポリシーに状態再構成特徴学習を組み込むことで、分布外状態への汎化性能を向上させる手法SRDPを提案し、実機ロボットとベンチマークで有効性を示した。
著者: Suzan Ece Ada, Erhan Oztop, Emre Ugur
分類: cs.LG, cs.AI, cs.NE, cs.RO
原文アブストラクト
Offline Reinforcement Learning (RL) methods leverage previous experiences to learn better policies than the behavior policy used for data collection. However, they face challenges handling distribution shifts due to the lack of online interaction during training. To this end, we propose a novel method named State Reconstruction for Diffusion Policies (SRDP) that incorporates state reconstruction feature learning in the recent class of diffusion policies to address the problem of out-of-distribution (OOD) generalization. Our method promotes learning of generalizable state representation to alleviate the distribution shift caused by OOD states. To illustrate the OOD generalization and faster convergence of SRDP, we design a novel 2D Multimodal Contextual Bandit environment and realize it on a 6-DoF real-world UR10 robot, as well as in simulation, and compare its performance with prior algorithms. In particular, we show the importance of the proposed state reconstruction via ablation studies. In addition, we assess the performance of our model on standard continuous control benchmarks (D4RL), namely the navigation of an 8-DoF ant and forward locomotion of half-cheetah, hopper, and walker2d, achieving state-of-the-art results. Finally, we demonstrate that our method can achieve 167% improvement over the competing baseline on a sparse continuous control navigation task where various regions of the state space are removed from the offline RL dataset, including the region encapsulating the goal.
関連論文
- FaST: オフライン強化学習のための遷移の因子分解合成オフライン強化学習
- 拡散ポリシーで誘導する拡散プランナーによるオフライン強化学習オフライン強化学習
- MAC-v2: ピクセル入力からの長期的モデルベース価値展開オフライン強化学習
- タスク認識表現を備えたDecision Transformerによるオフライン・メタ強化学習オフライン強化学習
- 効率的な拡散オフライン強化学習のための合成的平均速度モデリングオフライン強化学習
- 対話制約付きオフライン強化学習による自動運転オフライン強化学習
