マルチロボット強化学習による地図なしナビゲーションのためのDuelingネットワークで状態価値を集中化する手法
Centralizing State-Values in Dueling Networks for Multi-Robot Reinforcement Learning Mapless Navigation
CTDEパラダイムにおいて、集中型状態価値ネットワークでジョイント状態価値を計算し各エージェントの更新にグローバル状態情報を注入する新アーキテクチャを提案。2・4・8台のロボットナビゲーション実験でVDNやQMIXを上回る性能を示した。
著者: Enrico Marchesini, Alessandro Farinelli
分類: cs.MA, cs.AI, cs.LG, cs.RO
原文アブストラクト
We study the problem of multi-robot mapless navigation in the popular Centralized Training and Decentralized Execution (CTDE) paradigm. This problem is challenging when each robot considers its path without explicitly sharing observations with other robots and can lead to non-stationary issues in Deep Reinforcement Learning (DRL). The typical CTDE algorithm factorizes the joint action-value function into individual ones, to favor cooperation and achieve decentralized execution. Such factorization involves constraints (e.g., monotonicity) that limit the emergence of novel behaviors in an individual as each agent is trained starting from a joint action-value. In contrast, we propose a novel architecture for CTDE that uses a centralized state-value network to compute a joint state-value, which is used to inject global state information in the value-based updates of the agents. Consequently, each model computes its gradient update for the weights, considering the overall state of the environment. Our idea follows the insights of Dueling Networks as a separate estimation of the joint state-value has both the advantage of improving sample efficiency, while providing each robot information whether the global state is (or is not) valuable. Experiments in a robotic navigation task with 2 4, and 8 robots, confirm the superior performance of our approach over prior CTDE methods (e.g., VDN, QMIX).
関連論文
- マルチエージェントからマルチロボットへ:マルチロボット強化学習のためのスケーラブルな訓練・評価プラットフォームマルチロボット強化学習
- 内側注意機構による異種マルチロボット協調の適応的チーミングマルチロボット強化学習
- 集中Qネットで学習するマルチロボット分散マクロ行動ポリシーマルチロボット強化学習
