実世界画像からロボットシミュレーションタスクを生成するGRS
GRS: Generating Robotic Simulation Tasks from Real-World Images
単一のRGB-D画像からデジタルツインシミュレーションを構築し、仮想エージェント訓練用のタスクを自動生成するシステムを提案。
著者: Alex Zook, Fan-Yun Sun, Josef Spjut, Valts Blukis, Stan Birchfield, Jonathan Tremblay
分類: cs.RO, cs.AI
原文アブストラクト
We introduce GRS (Generating Robotic Simulation tasks), a system addressing real-to-sim for robotic simulations. GRS creates digital twin simulations from single RGB-D observations with solvable tasks for virtual agent training. Using vision-language models (VLMs), our pipeline operates in three stages: 1) scene comprehension with SAM2 for segmentation and object description, 2) matching objects with simulation-ready assets, and 3) generating appropriate tasks. We ensure simulation-task alignment through generated test suites and introduce a router that iteratively refines both simulation and test code. Experiments demonstrate our system's effectiveness in object correspondence and task environment generation through our novel router mechanism.
