後継特徴量を用いたリスク考慮型強化学習の転移
Risk-Aware Transfer in Reinforcement Learning using Successor Features
報酬の分散でリスクを測り、後継特徴量を拡張してリスクを考慮した方策転移を可能にする手法を提案し、ナビゲーションとロボットアーム制御で有効性を示した。
著者: Michael Gimelfarb, André Barreto, Scott Sanner, Chi-Guhn Lee
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
Sample efficiency and risk-awareness are central to the development of practical reinforcement learning (RL) for complex decision-making. The former can be addressed by transfer learning and the latter by optimizing some utility function of the return. However, the problem of transferring skills in a risk-aware manner is not well-understood. In this paper, we address the problem of risk-aware policy transfer between tasks in a common domain that differ only in their reward functions, in which risk is measured by the variance of reward streams. Our approach begins by extending the idea of generalized policy improvement to maximize entropic utilities, thus extending policy improvement via dynamic programming to sets of policies and levels of risk-aversion. Next, we extend the idea of successor features (SF), a value function representation that decouples the environment dynamics from the rewards, to capture the variance of returns. Our resulting risk-aware successor features (RaSF) integrate seamlessly within the RL framework, inherit the superior task generalization ability of SFs, and incorporate risk-awareness into the decision-making. Experiments on a discrete navigation domain and control of a simulated robotic arm demonstrate the ability of RaSFs to outperform alternative methods including SFs, when taking the risk of the learned policies into account.
