手術ロボットの長期的タスクのための価値情報付きスキル連鎖による方策学習
Value-Informed Skill Chaining for Policy Learning of Long-Horizon Tasks with Surgical Robot
手術ロボットの長期的タスクをサブタスクに分解し、状態価値関数を用いてサブタスク方策の接続に適した終了状態を選択することで、安定したスキル連鎖を実現する強化学習フレームワークを提案した。
著者: Tao Huang, Kai Chen, Wang Wei, Jianan Li, Yonghao Long, Qi Dou
分類: cs.RO, cs.AI, cs.LG
原文アブストラクト
Reinforcement learning is still struggling with solving long-horizon surgical robot tasks which involve multiple steps over an extended duration of time due to the policy exploration challenge. Recent methods try to tackle this problem by skill chaining, in which the long-horizon task is decomposed into multiple subtasks for easing the exploration burden and subtask policies are temporally connected to complete the whole long-horizon task. However, smoothly connecting all subtask policies is difficult for surgical robot scenarios. Not all states are equally suitable for connecting two adjacent subtasks. An undesired terminate state of the previous subtask would make the current subtask policy unstable and result in a failed execution. In this work, we introduce value-informed skill chaining (ViSkill), a novel reinforcement learning framework for long-horizon surgical robot tasks. The core idea is to distinguish which terminal state is suitable for starting all the following subtask policies. To achieve this target, we introduce a state value function that estimates the expected success probability of the entire task given a state. Based on this value function, a chaining policy is learned to instruct subtask policies to terminate at the state with the highest value so that all subsequent policies are more likely to be connected for accomplishing the task. We demonstrate the effectiveness of our method on three complex surgical robot tasks from SurRoL, a comprehensive surgical simulation platform, achieving high task success rates and execution efficiency. Code is available at $\href{https://github.com/med-air/ViSkill}{\text{https://github.com/med-air/ViSkill}}$.
