Touch-GS: 視覚と触覚で教師あり学習する3Dガウシアンスプラッティング
Touch-GS: Visual-Tactile Supervised 3D Gaussian Splatting
光学触覚センサと単眼深度推定を組み合わせ、触覚データを不確かさ付きで統合して3Dガウシアンスプラッティングを教師あり学習する手法を提案。透明・反射物体を含む少数視点シーン合成で視覚のみ・触覚のみより高精度を実現。
著者: Aiden Swann, Matthew Strong, Won Kyung Do, Gadiel Sznaier Camps, Mac Schwager, Monroe Kennedy
分類: cs.RO, cs.CV
原文アブストラクト
In this work, we propose a novel method to supervise 3D Gaussian Splatting (3DGS) scenes using optical tactile sensors. Optical tactile sensors have become widespread in their use in robotics for manipulation and object representation; however, raw optical tactile sensor data is unsuitable to directly supervise a 3DGS scene. Our representation leverages a Gaussian Process Implicit Surface to implicitly represent the object, combining many touches into a unified representation with uncertainty. We merge this model with a monocular depth estimation network, which is aligned in a two stage process, coarsely aligning with a depth camera and then finely adjusting to match our touch data. For every training image, our method produces a corresponding fused depth and uncertainty map. Utilizing this additional information, we propose a new loss function, variance weighted depth supervised loss, for training the 3DGS scene model. We leverage the DenseTact optical tactile sensor and RealSense RGB-D camera to show that combining touch and vision in this manner leads to quantitatively and qualitatively better results than vision or touch alone in a few-view scene syntheses on opaque as well as on reflective and transparent objects. Please see our project page at http://armlabstanford.github.io/touch-gs
