基盤モデルのフィードバックによる方策適応
Policy Adaptation from Foundation Model Feedback
訓練済み方策を新しいタスクや環境に展開する際、ランダムな指示で収集したデモを基盤モデルのフィードバックで再ラベリングし、自動的に微調整データを作る手法PAFFを提案。未知物体・タスク・環境やsim-to-realで大幅に性能を改善した。
著者: Yuying Ge, Annabella Macaluso, Li Erran Li, Ping Luo, Xiaolong Wang
分類: cs.LG, cs.AI, cs.CV, cs.RO
原文アブストラクト
Recent progress on vision-language foundation models have brought significant advancement to building general-purpose robots. By using the pre-trained models to encode the scene and instructions as inputs for decision making, the instruction-conditioned policy can generalize across different objects and tasks. While this is encouraging, the policy still fails in most cases given an unseen task or environment. In this work, we propose Policy Adaptation from Foundation model Feedback (PAFF). When deploying the trained policy to a new task or a new environment, we first let the policy play with randomly generated instructions to record the demonstrations. While the execution could be wrong, we can use the pre-trained foundation models to provide feedback to relabel the demonstrations. This automatically provides new pairs of demonstration-instruction data for policy fine-tuning. We evaluate our method on a broad range of experiments with the focus on generalization on unseen objects, unseen tasks, unseen environments, and sim-to-real transfer. We show PAFF improves baselines by a large margin in all cases. Our project page is available at https://geyuying.github.io/PAFF/
