要旨で参照されている関連研究として、拡散ポリシーを用いたBCベースの研究(Diffusion Policy)や、RLでのポリシー表現に関する研究が挙げられる。また、マルチタスク操作やゼロショット転移に関する研究も関連する。具体的には、拡散ポリシーの原論文(Chi et al.)や、RLと拡散モデルを組み合わせた研究、カリキュラム学習や目標指向表現を用いた研究が次に読むべき候補である。ただし、要旨に明示的な参照はないため、同分野の定番として、Diffusion Policy (Chi et al., 2023)や、RLでのカリキュラム学習、sim-to-real転移に関する論文が挙げられる。
Diffusion policies have shown promising empirical performance in representing and learning complex maneuvers for robots using behavior cloning (BC). In this paper, we explore training diffusion policies from scratch using reinforcement learning (RL) for multi-task robotic manipulation. Specifically, we aim to train a single diffusion policy for block-pushing tasks with multiple shapes. The proposed framework features a simple policy loss function, which is a reweighted evidence lower bound used in BC-based diffusion policy training and can seamlessly serve as the policy learning module in RL algorithms. To address the exploration challenges arising from the absence of demonstrations, we incorporate reverse curriculum generation and objective-centric representations. Combined with the expressiveness of diffusion policies, our design supports learning of multi-task block-pushing policies in our sparse-reward simulation setting. We further evaluate whether the trained diffusion policy transfers in zero-shot to real-world tasks under varying environmental conditions including goal positions, block shapes, block weights and surface friction, providing evidence that this pipeline can transfer to our real-world block-pushing setup under the tested variations.
