テキスト・視覚関連性と同変言語マッピングによる効率的で堅牢な言語条件付きマニピュレーションの学習
Learning Efficient and Robust Language-conditioned Manipulation using Textual-Visual Relevancy and Equivariant Language Mapping
事前学習済み視覚言語モデルと同変言語マッピングを組み合わせ、少ないロボットデータで言語指示に基づく操作を高精度かつ堅牢に実現する手法GEMを提案。
著者: Mingxi Jia, Haojie Huang, Zhewen Zhang, Chenghao Wang, Linfeng Zhao, Dian Wang, Jason Xinyu Liu, Robin Walters, Robert Platt, Stefanie Tellex
分類: cs.RO
原文アブストラクト
Controlling robots through natural language is pivotal for enhancing human-robot collaboration and synthesizing complex robot behaviors. Recent works that are trained on large robot datasets show impressive generalization abilities. However, such pretrained methods are (1) often fragile to unseen scenarios, and (2) expensive to adapt to new tasks. This paper introduces Grounded Equivariant Manipulation (GEM), a robust yet efficient approach that leverages pretrained vision-language models with equivariant language mapping for language-conditioned manipulation tasks. Our experiments demonstrate GEM's high sample efficiency and generalization ability across diverse tasks in both simulation and the real world. GEM achieves similar or higher performance with orders of magnitude fewer robot data compared with major data-efficient baselines such as CLIPort and VIMA. Finally, our approach demonstrates greater robustness compared to large VLA model, e.g, OpenVLA, at correctly interpreting natural language commands on unseen objects and poses. Code, data, and training details are available https://saulbatman.github.io/gem_page/
関連論文
- CALVIN：言語条件付き長期的ロボットマニピュレーションのベンチマーク言語条件付きマニピュレーション
- オフラインデータとクラウドソーシング注釈から言語条件付きロボット行動を学習言語条件付きマニピュレーション
- 自然言語指示をロボット操作用プログラムに変換する手法言語条件付きマニピュレーション
