物理モデルで導く生成ネットワークの特徴分離フレームワーク
Physics-informed Guided Disentanglement in Generative Networks
画像変換における物理現象(遮蔽や霧など)の絡み合いを解きほぐすため、微分可能・非微分可能な物理モデルやニューラルネットで誘導する3つの分離戦略を提案し、変換品質と制御性を大幅に向上させた。
著者: Fabio Pizzati, Pietro Cerri, Raoul de Charette
分類: cs.CV, cs.AI, cs.LG, cs.RO
原文アブストラクト
Image-to-image translation (i2i) networks suffer from entanglement effects in presence of physics-related phenomena in target domain (such as occlusions, fog, etc), lowering altogether the translation quality, controllability and variability. In this paper, we propose a general framework to disentangle visual traits in target images. Primarily, we build upon collection of simple physics models, guiding the disentanglement with a physical model that renders some of the target traits, and learning the remaining ones. Because physics allows explicit and interpretable outputs, our physical models (optimally regressed on target) allows generating unseen scenarios in a controllable manner. Secondarily, we show the versatility of our framework to neural-guided disentanglement where a generative network is used in place of a physical model in case the latter is not directly accessible. Altogether, we introduce three strategies of disentanglement being guided from either a fully differentiable physics model, a (partially) non-differentiable physics model, or a neural network. The results show our disentanglement strategies dramatically increase performances qualitatively and quantitatively in several challenging scenarios for image translation.
