LLMによる運転：説明可能な自動運転のための物体レベルベクトルモダリティの融合
Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving
物体レベルのベクトル数値データを事前学習済みLLMに統合し、運転状況の理解・質問応答・行動生成を行うマルチモーダル運転モデルを提案。10kシナリオからなる16万QAペアのデータセットも構築した。
著者: Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, Jamie Shotton
分類: cs.RO, cs.AI, cs.CL, cs.CV
原文アブストラクト
Large Language Models (LLMs) have shown promise in the autonomous driving sector, particularly in generalization and interpretability. We introduce a unique object-level multimodal LLM architecture that merges vectorized numeric modalities with a pre-trained LLM to improve context understanding in driving situations. We also present a new dataset of 160k QA pairs derived from 10k driving scenarios, paired with high quality control commands collected with RL agent and question answer pairs generated by teacher LLM (GPT-3.5). A distinct pretraining strategy is devised to align numeric vector modalities with static LLM representations using vector captioning language data. We also introduce an evaluation metric for Driving QA and demonstrate our LLM-driver's proficiency in interpreting driving scenarios, answering questions, and decision-making. Our findings highlight the potential of LLM-based driving action generation in comparison to traditional behavioral cloning. We make our benchmark, datasets, and model available for further exploration.
