Diff-2-in-1：拡散モデルによる生成と高密度知覚の橋渡し
Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models
拡散モデルの脱ノイズ過程を活用し、マルチモーダルデータ生成と高密度視覚知覚を同時に扱う統合フレームワークを提案。生成データを自己改善学習に用いることで、様々な知覚バックボーンの性能を向上させる。
著者: Shuhong Zheng, Zhipeng Bao, Ruoyu Zhao, Martial Hebert, Yu-Xiong Wang
分類: cs.CV, cs.LG, cs.RO
原文アブストラクト
Beyond high-fidelity image synthesis, diffusion models have recently exhibited promising results in dense visual perception tasks. However, most existing work treats diffusion models as a standalone component for perception tasks, employing them either solely for off-the-shelf data augmentation or as mere feature extractors. In contrast to these isolated and thus sub-optimal efforts, we introduce a unified, versatile, diffusion-based framework, Diff-2-in-1, that can simultaneously handle both multi-modal data generation and dense visual perception, through a unique exploitation of the diffusion-denoising process. Within this framework, we further enhance discriminative visual perception via multi-modal generation, by utilizing the denoising network to create multi-modal data that mirror the distribution of the original training set. Importantly, Diff-2-in-1 optimizes the utilization of the created diverse and faithful data by leveraging a novel self-improving learning mechanism. Comprehensive experimental evaluations validate the effectiveness of our framework, showcasing consistent performance improvements across various discriminative backbones and high-quality multi-modal data generation characterized by both realism and usefulness.
