人間のフィードバックを用いたオフライン強化学習の実環境展開
Deploying Offline Reinforcement Learning with Human Feedback
オフライン強化学習モデルを実環境に展開する際に、人間の監督とフィードバックを活用して安全性と性能を向上させる枠組みを提案し、ロボット歩行制御と交通信号制御で有効性を示した。
著者: Ziniu Li, Ke Xu, Liu Liu, Lanqing Li, Deheng Ye, Peilin Zhao
分類: cs.LG, cs.AI
原文アブストラクト
Reinforcement learning (RL) has shown promise for decision-making tasks in real-world applications. One practical framework involves training parameterized policy models from an offline dataset and subsequently deploying them in an online environment. However, this approach can be risky since the offline training may not be perfect, leading to poor performance of the RL models that may take dangerous actions. To address this issue, we propose an alternative framework that involves a human supervising the RL models and providing additional feedback in the online deployment phase. We formalize this online deployment problem and develop two approaches. The first approach uses model selection and the upper confidence bound algorithm to adaptively select a model to deploy from a candidate set of trained offline RL models. The second approach involves fine-tuning the model in the online deployment phase when a supervision signal arrives. We demonstrate the effectiveness of these approaches for robot locomotion control and traffic light control tasks through empirical validation.
