リンゴとオレンジの比較：LLMを活用した物体分類タスクにおけるマルチモーダル意図予測
Comparing Apples to Oranges: LLM-powered Multimodal Intention Prediction in an Object Categorization Task
物体分類タスクで協働するロボットが、ユーザーのジェスチャー・姿勢・表情・発話・環境状態をLLMで統合し、意図を階層的に予測する手法を提案・評価した。
著者: Hassan Ali, Philipp Allgeuer, Stefan Wermter
分類: cs.RO, cs.AI, cs.HC
原文アブストラクト
Human intention-based systems enable robots to perceive and interpret user actions to interact with humans and adapt to their behavior proactively. Therefore, intention prediction is pivotal in creating a natural interaction with social robots in human-designed environments. In this paper, we examine using Large Language Models (LLMs) to infer human intention in a collaborative object categorization task with a physical robot. We propose a novel multimodal approach that integrates user non-verbal cues, like hand gestures, body poses, and facial expressions, with environment states and user verbal cues to predict user intentions in a hierarchical architecture. Our evaluation of five LLMs shows the potential for reasoning about verbal and non-verbal user cues, leveraging their context-understanding and real-world knowledge to support intention prediction while collaborating on a task with a social robot. Video: https://youtu.be/tBJHfAuzohI
