NVFi: 動画から3D物理を学習するニューラル速度場
NVFi: Neural Velocity Fields for 3D Physics Learning from Dynamic Videos
多視点動画から3Dシーンの幾何・見た目・物理速度を同時に学習し、未来フレーム予測や動きの転移を可能にする手法を提案。
著者: Jinxi Li, Ziyang Song, Bo Yang
分類: cs.CV, cs.AI, cs.GR, cs.LG, cs.RO
原文アブストラクト
In this paper, we aim to model 3D scene dynamics from multi-view videos. Unlike the majority of existing works which usually focus on the common task of novel view synthesis within the training time period, we propose to simultaneously learn the geometry, appearance, and physical velocity of 3D scenes only from video frames, such that multiple desirable applications can be supported, including future frame extrapolation, unsupervised 3D semantic scene decomposition, and dynamic motion transfer. Our method consists of three major components, 1) the keyframe dynamic radiance field, 2) the interframe velocity field, and 3) a joint keyframe and interframe optimization module which is the core of our framework to effectively train both networks. To validate our method, we further introduce two dynamic 3D datasets: 1) Dynamic Object dataset, and 2) Dynamic Indoor Scene dataset. We conduct extensive experiments on multiple datasets, demonstrating the superior performance of our method over all baselines, particularly in the critical tasks of future frame extrapolation and unsupervised 3D semantic scene decomposition.
