モデルベース強化学習における教師なしダイナミクス汎化のための関係的介入アプローチ
A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning
環境ごとの潜在情報から冗長な情報を減らし、未知のダイナミクスを持つ環境へのゼロショット汎化を改善する手法を提案した論文。
著者: Jixian Guo, Mingming Gong, Dacheng Tao
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
The generalization of model-based reinforcement learning (MBRL) methods to environments with unseen transition dynamics is an important yet challenging problem. Existing methods try to extract environment-specified information $Z$ from past transition segments to make the dynamics prediction model generalizable to different dynamics. However, because environments are not labelled, the extracted information inevitably contains redundant information unrelated to the dynamics in transition segments and thus fails to maintain a crucial property of $Z$: $Z$ should be similar in the same environment and dissimilar in different ones. As a result, the learned dynamics prediction function will deviate from the true one, which undermines the generalization ability. To tackle this problem, we introduce an interventional prediction module to estimate the probability of two estimated $\hat{z}_i, \hat{z}_j$ belonging to the same environment. Furthermore, by utilizing the $Z$'s invariance within a single environment, a relational head is proposed to enforce the similarity between $\hat{{Z}}$ from the same environment. As a result, the redundant information will be reduced in $\hat{Z}$. We empirically show that $\hat{{Z}}$ estimated by our method enjoy less redundant information than previous methods, and such $\hat{{Z}}$ can significantly reduce dynamics prediction errors and improve the performance of model-based RL methods on zero-shot new environments with unseen dynamics. The codes of this method are available at \url{https://github.com/CR-Gjx/RIA}.
関連論文
- テスト時モデルベース強化学習による世界モデルへの記憶の符号化モデルベース強化学習
- 再考:世界モデルにおける比較的想像を通じた探索と計画モデルベース強化学習
- アクションジェット:行動条件付きデータ多様体の局所形状の学習モデルベース強化学習
- 未モデル状態依存性下での学習クリティックによる最適制御モデルベース強化学習
- VIGOR: モデルベース強化学習における潜在空間一貫性によるゼロショット視覚汎化モデルベース強化学習
- 行動なし時系列からの動的埋め込みによる転移可能な方策学習モデルベース強化学習
