LLMと複合現実による協働ロボットのウェイポイント生成
Enabling Waypoint Generation for Collaborative Robots using LLMs and Mixed Reality
自然言語で指示するだけでLLMが作業空間を理解しウェイポイントを生成、ARで結果を可視化する協働ロボット向けフレームワークを提案し、実機でピック&プレースを実証した。
著者: Cathy Mengying Fang, Krzysztof Zieliński, Pattie Maes, Joe Paradiso, Bruce Blumberg, Mikkel Baun Kjærgaard
分類: cs.HC, cs.RO
原文アブストラクト
Programming a robotic is a complex task, as it demands the user to have a good command of specific programming languages and awareness of the robot's physical constraints. We propose a framework that simplifies robot deployment by allowing direct communication using natural language. It uses large language models (LLM) for prompt processing, workspace understanding, and waypoint generation. It also employs Augmented Reality (AR) to provide visual feedback of the planned outcome. We showcase the effectiveness of our framework with a simple pick-and-place task, which we implement on a real robot. Moreover, we present an early concept of expressive robot behavior and skill generation that can be used to communicate with the user and learn new skills (e.g., object grasping).
