CLIP特徴量とランダム制御による画像・テキスト指示のマルチタスクロボット制御
CLIP feature-based randomized control using images and text for multiple tasks and robots
視覚言語モデルCLIPで画像と目標テキストの類似度を計算し、ランダム制御で類似度勾配を探索することで、追加学習なしに多様なタスク・ロボットを制御する手法を提案した。
著者: Kazuki Shibata, Hideki Deguchi, Shun Taguchi
分類: cs.RO
原文アブストラクト
This study presents a control framework leveraging vision language models (VLMs) for multiple tasks and robots. Notably, existing control methods using VLMs have achieved high performance in various tasks and robots in the training environment. However, these methods incur high costs for learning control policies for tasks and robots other than those in the training environment. Considering the application of industrial and household robots, learning in novel environments where robots are introduced is challenging. To address this issue, we propose a control framework that does not require learning control policies. Our framework combines the vision-language CLIP model with a randomized control. CLIP computes the similarity between images and texts by embedding them in the feature space. This study employs CLIP to compute the similarity between camera images and text representing the target state. In our method, the robot is controlled by a randomized controller that simultaneously explores and increases the similarity gradients. Moreover, we fine-tune the CLIP to improve the performance of the proposed method. Consequently, we confirm the effectiveness of our approach through a multitask simulation and a real robot experiment using a two-wheeled robot and robot arm.
