ETGL-DDPG: 疎な報酬の連続制御のための深層決定論的方策勾配アルゴリズム
ETGL-DDPG: A Deep Deterministic Policy Gradient Algorithm for Sparse Reward Continuous Control
疎な報酬の連続制御タスクにおいて、探索手法εt-greedy、二重経験再生バッファGDRB、最長nステップリターンをDDPGに統合し、標準ベンチマークで性能を向上させた。
著者: Ehsan Futuhi, Shayan Karimi, Chao Gao, Martin Müller
分類: cs.LG, cs.RO, stat.ML
原文アブストラクト
We consider deep deterministic policy gradient (DDPG) in the context of reinforcement learning with sparse rewards. To enhance exploration, we introduce a search procedure, \emph{${\epsilon}{t}$-greedy}, which generates exploratory options for exploring less-visited states. We prove that search using $\epsilon t$-greedy has polynomial sample complexity under mild MDP assumptions. To more efficiently use the information provided by rewarded transitions, we develop a new dual experience replay buffer framework, \emph{GDRB}, and implement \emph{longest n-step returns}. The resulting algorithm, \emph{ETGL-DDPG}, integrates all three techniques: \bm{$\epsilon t$}-greedy, \textbf{G}DRB, and \textbf{L}ongest $n$-step, into DDPG. We evaluate ETGL-DDPG on standard benchmarks and demonstrate that it outperforms DDPG, as well as other state-of-the-art methods, across all tested sparse-reward continuous environments. Ablation studies further highlight how each strategy individually enhances the performance of DDPG in this setting.
