階層的エンパワーメント：扱いやすいエンパワーメントベースのスキル学習に向けて
Hierarchical Empowerment: Towards Tractable Empowerment-Based Skill Learning
目標条件付き階層強化学習の概念を統合し、短期的なエンパワーメント計算のための新しい変分下界と、指数関数的に長い時間スケールでの階層的アーキテクチャを導入することで、エンパワーメントベースのスキル学習をより扱いやすくするフレームワークを提案。
著者: Andrew Levy, Sreehari Rammohan, Alessandro Allievi, Scott Niekum, George Konidaris
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
General purpose agents will require large repertoires of skills. Empowerment -- the maximum mutual information between skills and states -- provides a pathway for learning large collections of distinct skills, but mutual information is difficult to optimize. We introduce a new framework, Hierarchical Empowerment, that makes computing empowerment more tractable by integrating concepts from Goal-Conditioned Hierarchical Reinforcement Learning. Our framework makes two specific contributions. First, we introduce a new variational lower bound on mutual information that can be used to compute empowerment over short horizons. Second, we introduce a hierarchical architecture for computing empowerment over exponentially longer time scales. We verify the contributions of the framework in a series of simulated robotics tasks. In a popular ant navigation domain, our four level agents are able to learn skills that cover a surface area over two orders of magnitude larger than prior work.
