OpenESS: オープン語彙によるイベントベースの意味的シーン理解
OpenESS: Event-based Semantic Scene Understanding with Open Vocabularies
画像・テキスト・イベントデータを統合し、CLIPの知識をイベントストリームに転移することで、アノテーション不要でイベントカメラの意味的セグメンテーションを実現した。
著者: Lingdong Kong, Youquan Liu, Lai Xing Ng, Benoit R. Cottereau, Wei Tsang Ooi
分類: cs.CV, cs.RO
原文アブストラクト
Event-based semantic segmentation (ESS) is a fundamental yet challenging task for event camera sensing. The difficulties in interpreting and annotating event data limit its scalability. While domain adaptation from images to event data can help to mitigate this issue, there exist data representational differences that require additional effort to resolve. In this work, for the first time, we synergize information from image, text, and event-data domains and introduce OpenESS to enable scalable ESS in an open-world, annotation-efficient manner. We achieve this goal by transferring the semantically rich CLIP knowledge from image-text pairs to event streams. To pursue better cross-modality adaptation, we propose a frame-to-event contrastive distillation and a text-to-event semantic consistency regularization. Experimental results on popular ESS benchmarks showed our approach outperforms existing methods. Notably, we achieve 53.93% and 43.31% mIoU on DDD17 and DSEC-Semantic without using either event or frame labels.
関連論文
- 屋内散乱環境における物体セグメンテーションのためのニューロモーフィックデータセットイベントカメラ/セグメンテーション
