選好トランスフォーマー：人間の選好をトランスフォーマーでモデル化する強化学習
Preference Transformer: Modeling Human Preferences using Transformers for RL
人間の選好に基づく強化学習において、非マルコフ的報酬の重み付き和として選好をモデル化するトランスフォーマーアーキテクチャを提案し、実人間の選好を用いた制御タスクで従来法を上回る性能を示した。
著者: Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, Kimin Lee
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
Preference-based reinforcement learning (RL) provides a framework to train agents using human preferences between two behaviors. However, preference-based RL has been challenging to scale since it requires a large amount of human feedback to learn a reward function aligned with human intent. In this paper, we present Preference Transformer, a neural architecture that models human preferences using transformers. Unlike prior approaches assuming human judgment is based on the Markovian rewards which contribute to the decision equally, we introduce a new preference model based on the weighted sum of non-Markovian rewards. We then design the proposed preference model using a transformer architecture that stacks causal and bidirectional self-attention layers. We demonstrate that Preference Transformer can solve a variety of control tasks using real human preferences, while prior approaches fail to work. We also show that Preference Transformer can induce a well-specified reward and attend to critical events in the trajectory by automatically capturing the temporal dependencies in human decision-making. Code is available on the project website: https://sites.google.com/view/preference-transformer.
