TalkWithMachines：大規模・視覚言語モデルによる解釈可能な産業用ロボティクスのための人間-ロボットインタラクション強化
TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Models
LLMとVLMをロボットの知覚・制御に統合し、自然言語指示の理解や内部状態の言語化を通じて、安全が重要な産業用ロボットの解釈可能性と人間-ロボットインタラクションを向上させる4つの制御ワークフローを提案・実験した論文。
著者: Ammar N. Abbas, Csaba Beleznai
分類: cs.RO, cs.AI, cs.CL, cs.HC, cs.LG
原文アブストラクト
TalkWithMachines aims to enhance human-robot interaction by contributing to interpretable industrial robotic systems, especially for safety-critical applications. The presented paper investigates recent advancements in Large Language Models (LLMs) and Vision Language Models (VLMs), in combination with robotic perception and control. This integration allows robots to understand and execute commands given in natural language and to perceive their environment through visual and/or descriptive inputs. Moreover, translating the LLM's internal states and reasoning into text that humans can easily understand ensures that operators gain a clearer insight into the robot's current state and intentions, which is essential for effective and safe operation. Our paper outlines four LLM-assisted simulated robotic control workflows, which explore (i) low-level control, (ii) the generation of language-based feedback that describes the robot's internal states, (iii) the use of visual information as additional input, and (iv) the use of robot structure information for generating task plans and feedback, taking the robot's physical capabilities and limitations into account. The proposed concepts are presented in a set of experiments, along with a brief discussion. Project description, videos, and supplementary materials will be available on the project website: https://talk-machines.github.io.
