汎用フローを基盤としたスケーラブルなロボット学習のためのアフォーダンス
General Flow as Foundation Affordance for Scalable Robot Learning
大規模な人間のRGBD動画から言語条件付き3Dフローモデルを学習し、その予測を閉ループ方策に用いることで、追加学習なしに18タスクで81%のゼロショット人間→ロボット技能転移を実現した。
著者: Chengbo Yuan, Chuan Wen, Tong Zhang, Yang Gao
分類: cs.RO, cs.AI, cs.CV
原文アブストラクト
We address the challenge of acquiring real-world manipulation skills with a scalable framework. We hold the belief that identifying an appropriate prediction target capable of leveraging large-scale datasets is crucial for achieving efficient and universal learning. Therefore, we propose to utilize 3D flow, which represents the future trajectories of 3D points on objects of interest, as an ideal prediction target. To exploit scalable data resources, we turn our attention to human videos. We develop, for the first time, a language-conditioned 3D flow prediction model directly from large-scale RGBD human video datasets. Our predicted flow offers actionable guidance, thus facilitating zero-shot skill transfer in real-world scenarios. We deploy our method with a policy based on closed-loop flow prediction. Remarkably, without any in-domain finetuning, our method achieves an impressive 81\% success rate in zero-shot human-to-robot skill transfer, covering 18 tasks in 6 scenes. Our framework features the following benefits: (1) scalability: leveraging cross-embodiment data resources; (2) wide application: multiple object categories, including rigid, articulated, and soft bodies; (3) stable skill transfer: providing actionable guidance with a small inference domain-gap. Code, data, and supplementary materials are available https://general-flow.github.io
