HiL-ResRL: 人間参加型残差強化学習によるモデル非依存のファインチューニングアダプタ
HiL-ResRL: A Model-Agnostic Finetuning Adapter via Human-in-the-loop Residual Reinforcement Learning
模倣学習のVLAモデルに対して、モデル非依存でプラグイン可能な残差ポリシーを強化学習で訓練し、実ロボットで1.5時間の学習で成功率95%以上を達成するファインチューニング手法を提案した。
詳しい要約
1. どんなもの？
2. 先行研究と比べてどこがすごい？
3. 技術・手法の肝は？
4. どうやって有効だと検証した？
5. 議論はある？
6. 次に読むべき論文は？
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Jingyi Liu, Zhaohong Mai, ShunSen He, Hang Ren, Chao Wang, Shunbo Zhou, XiaoDong Wu, Heng Zhang
分類: cs.RO
原文アブストラクト
Recent advancements in generative imitation learning have significantly propelled the field of robotic manipulation. However, the majority of existing models rely heavily on Behavior Cloning (BC), a paradigm that suffers from compounding errors and distributional shift. Consequently, the efficacy of these models in practical industrial deployments remains limited. To address these challenges, we introduce a novel, plug-and-play fine-tuning pipeline designed to facilitate the robust deployment of Vision-Language-Action (VLA) models in real-world environments. In contrast to contemporary reinforcement learning (RL) fine-tuning strategies, which are often constrained by specific model architectures, our proposed framework is model-agnostic and adaptable to a diverse range of VLA models. We conceptualize VLA-generated actions as a unified interface, upon which we train a residual policy. This policy is designed to rectify suboptimal actions and address the distributional shift inherent in imitation learning. Additionally, we incorporate human-in-the-loop guidance to ensure safe exploration and maximize training efficiency. We conduct experiments directly in real-world robotic settings. The results demonstrate that within only 1.5 hour of real-world online RL training, the average success rate exceeds 95% on real robots. Our work presents a practical solution for deploying behavior cloning models in industrial scenarios.
関連論文
- Curie's Jarvis：印刷プロトコルからの自律ロボット化学実験ロボットマニピュレーション
- OMAR: 未知物体を推論して棚を整理するオープンワールド多属性推論ロボットマニピュレーション
- 編み物ロボット：深層学習による布地パターンの逆解析手法ロボットマニピュレーション
- 軌道最適化に導かれた敵対的正則化ポリシー学習ロボットマニピュレーション
