視覚言語モデルを活用したオープンボキャブラリ物体検出・追跡
Leveraging Vision-Language Models for Open-Vocabulary Instance Segmentation and Tracking
VLMの記述力とオープンボキャブラリ検出・動画セグメンテーションを組み合わせ、動的環境で未知物体をリアルタイムにセグメンテーション・追跡する手法を提案。
著者: Bastian Pätzold, Jan Nogga, Sven Behnke
分類: cs.CV, cs.RO
原文アブストラクト
Vision-language models (VLMs) excel in visual understanding but often lack reliable grounding capabilities and actionable inference rates. Integrating them with open-vocabulary object detection (OVD), instance segmentation, and tracking leverages their strengths while mitigating these drawbacks. We utilize VLM-generated structured descriptions to identify visible object instances, collect application-relevant attributes, and inform an open-vocabulary detector to extract corresponding bounding boxes that are passed to a video segmentation model providing segmentation masks and tracking. Once initialized, this model directly extracts segmentation masks, processing image streams in real time with minimal computational overhead. Tracks can be updated online as needed by generating new structured descriptions and detections. This combines the descriptive power of VLMs with the grounding capability of OVD and the pixel-level understanding and speed of video segmentation. Our evaluation across datasets and robotics platforms demonstrates the broad applicability of this approach, showcasing its ability to extract task-specific attributes from non-standard objects in dynamic environments. Code, data, videos, and benchmarks are available at https://vlm-gist.github.io
