深層強化学習による押し動作と把持の長期行動最適化
Optimizing the Long-Term Behaviour of Deep Reinforcement Learning for Pushing and Grasping
VPGとHourglassの長期報酬予測能力を新しいビン仕分けタスクで評価し、Double Q-Learningの組み合わせが高い割引率での長期行動系列予測に有効であることを示した。
著者: Rodrigo Chau
分類: cs.LG, cs.CV, cs.RO
原文アブストラクト
We investigate the "Visual Pushing for Grasping" (VPG) system by Zeng et al. and the "Hourglass" system by Ewerton et al., an evolution of the former. The focus of our work is the investigation of the capabilities of both systems to learn long-term rewards and policies. Zeng et al. original task only needs a limited amount of foresight. Ewerton et al. attain their best performance using an agent which only takes the most immediate action under consideration. We are interested in the ability of their models and training algorithms to accurately predict long-term Q-Values. To evaluate this ability, we design a new bin sorting task and reward function. Our task requires agents to accurately estimate future rewards and therefore use high discount factors in their Q-Value calculation. We investigate the behaviour of an adaptation of the VPG training algorithm on our task. We show that this adaptation can not accurately predict the required long-term action sequences. In addition to the limitations identified by Ewerton et al., it suffers from the known Deep Q-Learning problem of overestimated Q-Values. In an effort to solve our task, we turn to the Hourglass models and combine them with the Double Q-Learning approach. We show that this approach enables the models to accurately predict long-term action sequences when trained with large discount factors. Our results show that the Double Q-Learning technique is essential for training with very high discount factors, as the models Q-Value predictions diverge otherwise. We also experiment with different approaches for discount factor scheduling, loss calculation and exploration procedures. Our results show that the latter factors do not visibly influence the model's performance for our task.
関連論文
- タスク収束フローマッチングによる生成ロボティクスマニピュレーション
- 保持基準が視覚的リカバリ性能の評価を左右する:待機制御下での協調リカバリ分析マニピュレーション
- モデル情報を活用したロバスト幾何DDPCによるフィードバック線形化軌道追従マニピュレーション
- 学習済み力学系を用いた協調モバイルマニピュレーションマニピュレーション
- 内視鏡環境におけるMoveItを用いた逆運動学の改善マニピュレーション
- 複数の安定目標を持つ動作プリミティブを模倣学習するmulti-CONDORマニピュレーション
