反復的視覚プロンプティングによるオープンボキャブラリ動作ローカリゼーション
Open-Vocabulary Action Localization with Iterative Visual Prompting
学習不要で任意の動作を扱えるよう、視覚言語モデルにフレーム番号付きの連結画像を与え、サンプリング範囲を反復的に絞り込むことで動画中の動作区間を高精度に特定する手法を提案。
著者: Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, Katsushi Ikeuchi
分類: cs.CV, cs.AI, cs.RO
原文アブストラクト
Video action localization aims to find the timings of specific actions from a long video. Although existing learning-based approaches have been successful, they require annotating videos, which comes with a considerable labor cost. This paper proposes a training-free, open-vocabulary approach based on emerging off-the-shelf vision-language models (VLMs). The challenge stems from the fact that VLMs are neither designed to process long videos nor tailored for finding actions. We overcome these problems by extending an iterative visual prompting technique. Specifically, we sample video frames and create a concatenated image with frame index labels, allowing a VLM to identify the frames that most likely correspond to the start and end of the action. By iteratively narrowing the sampling window around the selected frames, the estimation gradually converges to more precise temporal boundaries. We demonstrate that this technique yields reasonable performance, achieving results comparable to state-of-the-art zero-shot action localization. These results support the use of VLMs as a practical tool for understanding videos. Sample code is available at https://microsoft.github.io/VLM-Video-Action-Localization/
