オフライン強化学習のための行動選好回帰
Behavior Preference Regression for Offline Reinforcement Learning
ペア比較に基づく選好最適化をオフライン強化学習に応用し、Q関数の最大モードを行動方策との整合性を保ちながら学習するBPRを提案。D4RLやV-D4RLで最先端性能を達成。
著者: Padmanaba Srinivasan, William Knottenbelt
分類: cs.LG, cs.AI
原文アブストラクト
Offline reinforcement learning (RL) methods aim to learn optimal policies with access only to trajectories in a fixed dataset. Policy constraint methods formulate policy learning as an optimization problem that balances maximizing reward with minimizing deviation from the behavior policy. Closed form solutions to this problem can be derived as weighted behavioral cloning objectives that, in theory, must compute an intractable partition function. Reinforcement learning has gained popularity in language modeling to align models with human preferences; some recent works consider paired completions that are ranked by a preference model following which the likelihood of the preferred completion is directly increased. We adapt this approach of paired comparison. By reformulating the paired-sample optimization problem, we fit the maximum-mode of the Q function while maximizing behavioral consistency of policy actions. This yields our algorithm, Behavior Preference Regression for offline RL (BPR). We empirically evaluate BPR on the widely used D4RL Locomotion and Antmaze datasets, as well as the more challenging V-D4RL suite, which operates in image-based state spaces. BPR demonstrates state-of-the-art performance over all domains. Our on-policy experiments suggest that BPR takes advantage of the stability of on-policy value functions with minimal perceptible performance degradation on Locomotion datasets.
関連論文
- FaST: オフライン強化学習のための遷移の因子分解合成オフライン強化学習
- 拡散ポリシーで誘導する拡散プランナーによるオフライン強化学習オフライン強化学習
- MAC-v2: ピクセル入力からの長期的モデルベース価値展開オフライン強化学習
- タスク認識表現を備えたDecision Transformerによるオフライン・メタ強化学習オフライン強化学習
- 効率的な拡散オフライン強化学習のための合成的平均速度モデリングオフライン強化学習
- 対話制約付きオフライン強化学習による自動運転オフライン強化学習
