行動系列を用いた粗から細へのQネットワークによるデータ効率的強化学習
Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learning
行動系列を入力としてQ値を予測するクリティックを学習する強化学習手法CQN-ASを提案し、疎な報酬のヒューマノイド制御や卓上操作タスクでデータ効率と性能を向上させた。
著者: Younggyo Seo, Pieter Abbeel
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
Predicting a sequence of actions has been crucial in the success of recent behavior cloning algorithms in robotics. Can similar ideas improve reinforcement learning (RL)? We answer affirmatively by observing that incorporating action sequences when predicting ground-truth return-to-go leads to lower validation loss. Motivated by this, we introduce Coarse-to-fine Q-Network with Action Sequence (CQN-AS), a novel value-based RL algorithm that learns a critic network that outputs Q-values over a sequence of actions, i.e., explicitly training the value function to learn the consequence of executing action sequences. Our experiments show that CQN-AS outperforms several baselines on a variety of sparse-reward humanoid control and tabletop manipulation tasks from BiGym and RLBench.
