追跡・修復・再スプラット:被写体駆動型3D/4D生成のための漸進的テクスチャ補完
Track, Inpaint, Resplat: Subject-driven 3D and 4D Generation with Progressive Texture Infilling
既存の3D生成モデルが作った3Dアセットに対し、動画トラッキングで修正領域を特定し、被写体駆動型の2Dインペインティングで段階的に補完した後、再スプラッティングで3Dに戻すことで、特定被写体の同一性を保った3D/4D生成を実現する手法を提案。
詳しい要約
1. どんなもの?
2. 先行研究と比べてどこがすごい?
3. 技術・手法の肝は?
4. どうやって有効だと検証した?
5. 議論はある?
6. 次に読むべき論文は?
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Shuhong Zheng, Ashkan Mirzaei, Igor Gilitschenski
分類: cs.CV, cs.AI, cs.GR, cs.LG, cs.RO
原文アブストラクト
Current 3D/4D generation methods are usually optimized for photorealism, efficiency, and aesthetics. However, they often fail to preserve the semantic identity of the subject across different viewpoints. Adapting generation methods with one or few images of a specific subject (also known as Personalization or Subject-driven generation) allows generating visual content that align with the identity of the subject. However, personalized 3D/4D generation is still largely underexplored. In this work, we introduce TIRE (Track, Inpaint, REsplat), a novel method for subject-driven 3D/4D generation. It takes an initial 3D asset produced by an existing 3D generative model as input and uses video tracking to identify the regions that need to be modified. Then, we adopt a subject-driven 2D inpainting model for progressively infilling the identified regions. Finally, we resplat the modified 2D multi-view observations back to 3D while still maintaining consistency. Extensive experiments demonstrate that our approach significantly improves identity preservation in 3D/4D generation compared to state-of-the-art methods. Our project website is available at https://zsh2000.github.io/track-inpaint-resplat.github.io/.
