新規性に基づくサンプル再利用による連続ロボティクス制御
Novelty-based Sample Reuse for Continuous Robotics Control
強化学習において、頻繁に観測される状態は更新を減らし、稀な新規状態には追加更新を行うことで、サンプルを効率的に再利用し、収束速度と成功率を向上させる手法NSRを提案した。
著者: Ke Duan, Kai Yang, Houde Liu, Xueqian Wang
分類: cs.RO, cs.LG
原文アブストラクト
In reinforcement learning, agents collect state information and rewards through environmental interactions, essential for policy refinement. This process is notably time-consuming, especially in complex robotic simulations and real-world applications. Traditional algorithms usually re-engage with the environment after processing a single batch of samples, thereby failing to fully capitalize on historical data. However, frequently observed states, with reliable value estimates, require minimal updates; in contrast, rare observed states necessitate more intensive updates for achieving accurate value estimations. To address uneven sample utilization, we propose Novelty-guided Sample Reuse (NSR). NSR provides extra updates for infrequent, novel states and skips additional updates for frequent states, maximizing sample use before interacting with the environment again. Our experiments show that NSR improves the convergence rate and success rate of algorithms without significantly increasing time consumption. Our code is publicly available at https://github.com/ppksigs/NSR-DDPG-HER.
