OTTER: テキスト認識型視覚特徴抽出を備えた視覚-言語-行動モデル
OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction
言語指示に意味的に対応する視覚特徴だけを選択的に抽出して行動予測に用いるVLAアーキテクチャを提案し、事前学習済みの視覚言語モデルを凍結したまま高いゼロショット汎化を実現した。
著者: Huang Huang, Fangchen Liu, Letian Fu, Tingfan Wu, Mustafa Mukadam, Jitendra Malik, Ken Goldberg, Pieter Abbeel
分類: cs.RO, cs.CV
原文アブストラクト
Vision-Language-Action (VLA) models aim to predict robotic actions based on visual observations and language instructions. Existing approaches require fine-tuning pre-trained visionlanguage models (VLMs) as visual and language features are independently fed into downstream policies, degrading the pre-trained semantic alignments. We propose OTTER, a novel VLA architecture that leverages these existing alignments through explicit, text-aware visual feature extraction. Instead of processing all visual features, OTTER selectively extracts and passes only task-relevant visual features that are semantically aligned with the language instruction to the policy transformer. This allows OTTER to keep the pre-trained vision-language encoders frozen. Thereby, OTTER preserves and utilizes the rich semantic understanding learned from large-scale pre-training, enabling strong zero-shot generalization capabilities. In simulation and real-world experiments, OTTER significantly outperforms existing VLA models, demonstrating strong zeroshot generalization to novel objects and environments. Video, code, checkpoints, and dataset: https://ottervla.github.io/.
