選好学習による報酬関数の特徴量獲得
Preference-based Learning of Reward Function Features
人間の比較データから報酬関数を学習する際、手設計の特徴量に加えてニューラルネットで追加特徴量も同時に学習し、運転シナリオで予測性能と表現力の向上を示した論文。
著者: Sydney M. Katz, Amir Maleki, Erdem Bıyık, Mykel J. Kochenderfer
分類: cs.RO, cs.HC
原文アブストラクト
Preference-based learning of reward functions, where the reward function is learned using comparison data, has been well studied for complex robotic tasks such as autonomous driving. Existing algorithms have focused on learning reward functions that are linear in a set of trajectory features. The features are typically hand-coded, and preference-based learning is used to determine a particular user's relative weighting for each feature. Designing a representative set of features to encode reward is challenging and can result in inaccurate models that fail to model the users' preferences or perform the task properly. In this paper, we present a method to learn both the relative weighting among features as well as additional features that help encode a user's reward function. The additional features are modeled as a neural network that is trained on the data from pairwise comparison queries. We apply our methods to a driving scenario used in previous work and compare the predictive power of our method to that of only hand-coded features. We perform additional analysis to interpret the learned features and examine the optimal trajectories. Our results show that adding an additional learned feature to the reward model enhances both its predictive power and expressiveness, producing unique results for each user.
