スキル選好：人間フィードバックからロボットスキルを抽出・実行する学習
Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback
人間の選好モデルを学習してオフラインデータから人間の意図に沿ったスキルを抽出し、さらに人間フィードバックを用いた強化学習で複雑な多段階操作タスクを解く手法SkiPを提案した。
著者: Xiaofei Wang, Kimin Lee, Kourosh Hakhamaneshi, Pieter Abbeel, Michael Laskin
分類: cs.LG, cs.RO
原文アブストラクト
A promising approach to solving challenging long-horizon tasks has been to extract behavior priors (skills) by fitting generative models to large offline datasets of demonstrations. However, such generative models inherit the biases of the underlying data and result in poor and unusable skills when trained on imperfect demonstration data. To better align skill extraction with human intent we present Skill Preferences (SkiP), an algorithm that learns a model over human preferences and uses it to extract human-aligned skills from offline data. After extracting human-preferred skills, SkiP also utilizes human feedback to solve down-stream tasks with RL. We show that SkiP enables a simulated kitchen robot to solve complex multi-step manipulation tasks and substantially outperforms prior leading RL algorithms with human preferences as well as leading skill extraction algorithms without human preferences.
