PointAction: 3D点群をロボット制御の普遍的な行動表現として用いる
PointAction: 3D Points as Universal Action Representations for Robot Control
ビデオ生成モデルを基盤に、将来のRGBフレームと動的3D点群を同時予測し、その点群の動きを行動デコーダでロボットの実行可能な行動に変換するフレームワークを提案。RGBのみの行動基盤の曖昧さを低減し、限られた行動教師データで異なるタスクやロボットへの転移を実現。
詳しい要約
1. どんなもの?
2. 先行研究と比べてどこがすごい?
3. 技術・手法の肝は?
4. どうやって有効だと検証した?
5. 議論はある?
6. 次に読むべき論文は?
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Mutian Tong, Han Jiang, Qiao Feng, Lingjie Liu, Jiatao Gu
分類: cs.RO, cs.CV, cs.LG
原文アブストラクト
Video-Action Models (VAMs) leverage the broad visual dynamics captured by pre-trained video diffusion models, offering a promising path toward generalizable robot manipulation. However, RGB-only video rollouts are not directly actionable: they leave metric 3D motion, contact geometry, and fine-grained spatial constraints under-specified, making action grounding ambiguous. Meanwhile, scaling action supervision across diverse tasks and embodiments remains costly. We present PointAction, a framework that bridges video predictions to robot actions through explicit point-based 4D modeling. PointAction fine-tunes a foundation video generation model to jointly predict future RGB frames and dynamic 3D pointmaps, producing temporally consistent 3D motion of task-relevant scene geometry. These point dynamics serve as a structured, embodiment-agnostic action interface, which a diffusion-based action decoder maps to executable robot actions. By using metric 3D point dynamics as the interface between video prediction and control, PointAction reduces the ambiguity of RGB-only action grounding and supports transfer across tasks and embodiments with limited action supervision. Experiments show that PointAction achieves state-of-the-art 4D generation quality on robot scenes, outperforms existing baselines in simulation, and generalizes to two real robot arms unseen during pretraining.
