再学習なしで凍結視覚言語モデルを拡張:ロボット知覚の改善に向けて
Expanding Frozen Vision-Language Models without Retraining: Towards Improved Robot Perception
IMUデータの埋め込みを視覚埋め込み空間に整列させることで、再学習なしに視覚言語モデルが追加モダリティを理解・推論できるようにする手法を提案し、人間活動認識実験で有効性を示した。
著者: Riley Tavassoli, Mani Amani, Reza Akhavian
分類: cs.AI, cs.RO
原文アブストラクト
Vision-language models (VLMs) have shown powerful capabilities in visual question answering and reasoning tasks by combining visual representations with the abstract skill set large language models (LLMs) learn during pretraining. Vision, while the most popular modality to augment LLMs with, is only one representation of a scene. In human-robot interaction scenarios, robot perception requires accurate scene understanding by the robot. In this paper, we define and demonstrate a method of aligning the embedding spaces of different modalities (in this case, inertial measurement unit (IMU) data) to the vision embedding space through a combination of supervised and contrastive training, enabling the VLM to understand and reason about these additional modalities without retraining. We opt to give the model IMU embeddings directly over using a separate human activity recognition model that feeds directly into the prompt to allow for any nonlinear interactions between the query, image, and IMU signal that would be lost by mapping the IMU data to a discrete activity label. Further, we demonstrate our methodology's efficacy through experiments involving human activity recognition using IMU data and visual inputs. Our results show that using multiple modalities as input improves the VLM's scene understanding and enhances its overall performance in various tasks, thus paving the way for more versatile and capable language models in multi-modal contexts.
