ManipGPT：大規模視覚モデルのアフォーダンスセグメンテーションは関節物体操作に十分か？
ManipGPT: Is Affordance Segmentation by Large Vision Models Enough for Articulated Object Manipulation?
大規模視覚Transformerを微調整し、関節物体の操作に必要な部位レベルのアフォーダンスマスクを予測する枠組みを提案。シミュレーションと実世界の画像データセットで学習し、インピーダンス適応方策と組み合わせることで複雑な知覚システムなしに操作を実現した。
著者: Taewhan Kim, Hojin Bae, Zeming Li, Xiaoqi Li, Iaroslav Ponomarenko, Ruihai Wu, Hao Dong
分類: cs.RO, cs.CV
原文アブストラクト
Visual actionable affordance has emerged as a transformative approach in robotics, focusing on perceiving interaction areas prior to manipulation. Traditional methods rely on pixel sampling to identify successful interaction samples or processing pointclouds for affordance mapping. However, these approaches are computationally intensive and struggle to adapt to diverse and dynamic environments. This paper introduces ManipGPT, a framework designed to predict optimal interaction areas for articulated objects using a large pre-trained vision transformer (ViT). We create a dataset of 9.9k simulated and real images to bridge the visual sim-to-real gap and enhance real-world applicability. By fine-tuning the vision transformer on this small dataset, we significantly improve part-level affordance segmentation, adapting the model's in-context segmentation capabilities to robot manipulation scenarios. This enables effective manipulation across simulated and real-world environments by generating part-level affordance masks, paired with an impedance adaptation policy, sufficiently eliminating the need for complex datasets or perception systems.
