マルチモーダル実演学習による人間とロボットの行動アラインメント
Toward Aligning Human and Robot Actions via Multi-Modal Demonstration Learning
RGB動画の人間実演とボクセル化RGB-D空間のロボット実演を統合的にモデル化し、pick and placeタスクで人間意図とロボット行動の対応を学習する枠組みを提案した。
著者: Azizul Zahid, Jie Fan, Farong Wang, Ashton Dy, Sai Swaminathan, Fei Liu
分類: cs.RO, cs.AI, cs.CV
原文アブストラクト
Understanding action correspondence between humans and robots is essential for evaluating alignment in decision-making, particularly in human-robot collaboration and imitation learning within unstructured environments. We propose a multimodal demonstration learning framework that explicitly models human demonstrations from RGB video with robot demonstrations in voxelized RGB-D space. Focusing on the "pick and place" task from the RH20T dataset, we utilize data from 5 users across 10 diverse scenes. Our approach combines ResNet-based visual encoding for human intention modeling and a Perceiver Transformer for voxel-based robot action prediction. After 2000 training epochs, the human model reaches 71.67% accuracy, and the robot model achieves 71.8% accuracy, demonstrating the framework's potential for aligning complex, multimodal human and robot behaviors in manipulation tasks.
