物理シーンの視覚基盤モデルに向けて
Towards Visual Foundational Models of Physical Scenes
画像予測のみを学習基準として物理シーンの汎用視覚表現を学習する第一歩を提案し、NeRFと拡散モデルを組み合わせたNeRF Diffusionを非教師あり表現として検証した。
著者: Chethan Parameshwara, Alessandro Achille, Matthew Trager, Xiaolong Li, Jiawei Mo, Matthew Trager, Ashwin Swaminathan, CJ Taylor, Dheera Venkatraman, Xiaohan Fei, Stefano Soatto
分類: cs.CV, cs.AI, cs.LG, cs.RO
原文アブストラクト
We describe a first step towards learning general-purpose visual representations of physical scenes using only image prediction as a training criterion. To do so, we first define "physical scene" and show that, even though different agents may maintain different representations of the same scene, the underlying physical scene that can be inferred is unique. Then, we show that NeRFs cannot represent the physical scene, as they lack extrapolation mechanisms. Those, however, could be provided by Diffusion Models, at least in theory. To test this hypothesis empirically, NeRFs can be combined with Diffusion Models, a process we refer to as NeRF Diffusion, used as unsupervised representations of the physical scene. Our analysis is limited to visual data, without external grounding mechanisms that can be provided by independent sensory modalities.
関連論文
- IronViT: 効率的な汎用視覚表現学習に向けて視覚基盤モデル
