一度だけ評価せよ:オフライン強化学習のためのシンプルなベースライン
You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
オフライン強化学習において、ポリシー評価を一度だけ行う単純なベースラインを提案し、D4RLベンチマークで競争力のある性能を示した。
著者: Wonjoon Goo, Scott Niekum
分類: cs.LG, cs.RO
原文アブストラクト
The goal of offline reinforcement learning (RL) is to find an optimal policy given prerecorded trajectories. Many current approaches customize existing off-policy RL algorithms, especially actor-critic algorithms in which policy evaluation and improvement are iterated. However, the convergence of such approaches is not guaranteed due to the use of complex non-linear function approximation and an intertwined optimization process. By contrast, we propose a simple baseline algorithm for offline RL that only performs the policy evaluation step once so that the algorithm does not require complex stabilization schemes. Since the proposed algorithm is not likely to converge to an optimal policy, it is an appropriate baseline for actor-critic algorithms that ought to be outperformed if there is indeed value in iterative optimization in the offline setting. Surprisingly, we empirically find that the proposed algorithm exhibits competitive and sometimes even state-of-the-art performance in a subset of the D4RL offline RL benchmark. This result suggests that future work is needed to fully exploit the potential advantages of iterative optimization in order to justify the reduced stability of such methods.
関連論文
- 未来表現を予測して意思決定するDT-JEPAオフラインRL
- 学習された時間的抽象化による確率的環境でのスケーラブルな意思決定オフラインRL
- 同変オフライン強化学習オフラインRL
- ベイズ再パラメータ化による報酬条件付き強化学習のエネルギー基盤モデルオフラインRL
- エージェント制御表現:豊富な外生情報を用いた原理的なオフライン強化学習オフラインRL
- ロボットのための事前学習:オフラインRLで少数試行から新タスクを学習オフラインRL
