深層強化学習のための統合型適応T-soft更新則
Consolidated Adaptive T-soft Update for Deep Reinforcement Learning
ターゲットネットワークの更新則T-softをAdaTermの更新則で適応化し、メインネットワークをターゲットへ引き戻す統合機構を加えたCAT-soft更新を提案・検証した論文。
分類: cs.LG, cs.RO
原文アブストラクト
Demand for deep reinforcement learning (DRL) is gradually increased to enable robots to perform complex tasks, while DRL is known to be unstable. As a technique to stabilize its learning, a target network that slowly and asymptotically matches a main network is widely employed to generate stable pseudo-supervised signals. Recently, T-soft update has been proposed as a noise-robust update rule for the target network and has contributed to improving the DRL performance. However, the noise robustness of T-soft update is specified by a hyperparameter, which should be tuned for each task, and is deteriorated by a simplified implementation. This study develops adaptive T-soft (AT-soft) update by utilizing the update rule in AdaTerm, which has been developed recently. In addition, the concern that the target network does not asymptotically match the main network is mitigated by a new consolidation for bringing the main network back to the target network. This so-called consolidated AT-soft (CAT-soft) update is verified through numerical simulations.
