PlaySlot: 制御可能な物体中心動画予測と計画のための逆潜在ダイナミクス学習
PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
ラベルなし動画から物体表現と潜在行動を推論し、将来の物体状態や動画フレームを予測する物体中心モデルを提案。潜在行動を条件に多様な未来を生成でき、ロボット行動学習にも活用可能。
著者: Angel Villar-Corrales, Sven Behnke
分類: cs.CV, cs.RO
原文アブストラクト
Predicting future scene representations is a crucial task for enabling robots to understand and interact with the environment. However, most existing methods rely on videos and simulations with precise action annotations, limiting their ability to leverage the large amount of available unlabeled video data. To address this challenge, we propose PlaySlot, an object-centric video prediction model that infers object representations and latent actions from unlabeled video sequences. It then uses these representations to forecast future object states and video frames. PlaySlot allows the generation of multiple possible futures conditioned on latent actions, which can be inferred from video dynamics, provided by a user, or generated by a learned action policy, thus enabling versatile and interpretable world modeling. Our results show that PlaySlot outperforms both stochastic and object-centric baselines for video prediction across different environments. Furthermore, we show that our inferred latent actions can be used to learn robot behaviors sample-efficiently from unlabeled video demonstrations. Videos and code are available on https://play-slot.github.io/PlaySlot/.
