部分観測マルコフ決定過程に対する時相論理制約付き強化学習
Reinforcement Learning with Temporal Logic Constraints for Partially-Observable Markov Decision Processes
未知の部分観測環境で時間依存の安全制約を満たす制御器を、信念MDPと時相論理の積構成を用いた強化学習で学習する手法を提案した論文。
著者: Yu Wang, Alper Kamil Bozkurt, Miroslav Pajic
分類: cs.RO
原文アブストラクト
This paper proposes a reinforcement learning method for controller synthesis of autonomous systems in unknown and partially-observable environments with subjective time-dependent safety constraints. Mathematically, we model the system dynamics by a partially-observable Markov decision process (POMDP) with unknown transition/observation probabilities. The time-dependent safety constraint is captured by iLTL, a variation of linear temporal logic for state distributions. Our Reinforcement learning method first constructs the belief MDP of the POMDP, capturing the time evolution of estimated state distributions. Then, by building the product belief MDP of the belief MDP and the limiting deterministic B\uchi automaton (LDBA) of the temporal logic constraint, we transform the time-dependent safety constraint on the POMDP into a state-dependent constraint on the product belief MDP. Finally, we learn the optimal policy by value iteration under the state-dependent constraint.
