ウォール街木探索:オフライン強化学習のためのリスク認識型計画法
Wall Street Tree Search: Risk-Aware Planning for Offline Reinforcement Learning
金融の現代ポートフォリオ理論を強化学習に応用し、不確実性によるリスクを考慮しながら行動を選ぶオフラインRLの計画アルゴリズムを提案。Transformerと組み合わせて最先端の性能を達成した。
著者: Dan Elbaz, Gal Novik, Oren Salzman
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
Offline reinforcement-learning (RL) algorithms learn to make decisions using a given, fixed training dataset without online data collection. This problem setting is captivating because it holds the promise of utilizing previously collected datasets without any costly or risky interaction with the environment. However, this promise also bears the drawback of this setting as the restricted dataset induces uncertainty because the agent can encounter unfamiliar sequences of states and actions that the training data did not cover. To mitigate the destructive uncertainty effects, we need to balance the aspiration to take reward-maximizing actions with the incurred risk due to incorrect ones. In financial economics, modern portfolio theory (MPT) is a method that risk-averse investors can use to construct diversified portfolios that maximize their returns without unacceptable levels of risk. We propose integrating MPT into the agent's decision-making process, presenting a new simple-yet-highly-effective risk-aware planning algorithm for offline RL. Our algorithm allows us to systematically account for the \emph{estimated quality} of specific actions and their \emph{estimated risk} due to the uncertainty. We show that our approach can be coupled with the Transformer architecture to yield a state-of-the-art planner, which maximizes the return for offline RL tasks. Moreover, our algorithm reduces the variance of the results significantly compared to conventional Transformer decoding, which results in a much more stable algorithm -- a property that is essential for the offline RL setting, where real-world exploration and failures can be costly or dangerous.
関連論文
- FaST: オフライン強化学習のための遷移の因子分解合成オフライン強化学習
- 拡散ポリシーで誘導する拡散プランナーによるオフライン強化学習オフライン強化学習
- MAC-v2: ピクセル入力からの長期的モデルベース価値展開オフライン強化学習
- タスク認識表現を備えたDecision Transformerによるオフライン・メタ強化学習オフライン強化学習
- 効率的な拡散オフライン強化学習のための合成的平均速度モデリングオフライン強化学習
- 対話制約付きオフライン強化学習による自動運転オフライン強化学習
