GestLLM：大規模言語モデルによる高度なハンドジェスチャー解釈とヒューマンロボットインタラクション
GestLLM: Advanced Hand Gesture Interpretation via Large Language Models for Human-Robot Interaction
MediaPipeで手の特徴を抽出し大規模言語モデルで解釈することで、事前定義に縛られない多様なジェスチャーでロボットを直感的に操作できるシステムを提案した。
著者: Oleg Kobzarev, Artem Lykov, Dzmitry Tsetserukou
分類: cs.RO
原文アブストラクト
This paper introduces GestLLM, an advanced system for human-robot interaction that enables intuitive robot control through hand gestures. Unlike conventional systems, which rely on a limited set of predefined gestures, GestLLM leverages large language models and feature extraction via MediaPipe to interpret a diverse range of gestures. This integration addresses key limitations in existing systems, such as restricted gesture flexibility and the inability to recognize complex or unconventional gestures commonly used in human communication. By combining state-of-the-art feature extraction and language model capabilities, GestLLM achieves performance comparable to leading vision-language models while supporting gestures underrepresented in traditional datasets. For example, this includes gestures from popular culture, such as the ``Vulcan salute" from Star Trek, without any additional pretraining, prompt engineering, etc. This flexibility enhances the naturalness and inclusivity of robot control, making interactions more intuitive and user-friendly. GestLLM provides a significant step forward in gesture-based interaction, enabling robots to understand and respond to a wide variety of hand gestures effectively. This paper outlines its design, implementation, and evaluation, demonstrating its potential applications in advanced human-robot collaboration, assistive robotics, and interactive entertainment.
