STEVE-Audio: Minecraftエージェントの目標条件付けモダリティの拡張
STEVE-Audio: Expanding the Goal Conditioning Modalities of Embodied Agents in Minecraft
MinecraftエージェントSTEVE-1に音声を目標条件として与えられるよう拡張し、音声から潜在目標空間へのマッピングを学習した研究。
著者: Nicholas Lenzen, Amogh Raut, Andrew Melnik
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
Recently, the STEVE-1 approach has been introduced as a method for training generative agents to follow instructions in the form of latent CLIP embeddings. In this work, we present a methodology to extend the control modalities by learning a mapping from new input modalities to the latent goal space of the agent. We apply our approach to the challenging Minecraft domain, and extend the goal conditioning to include the audio modality. The resulting audio-conditioned agent is able to perform on a comparable level to the original text-conditioned and visual-conditioned agents. Specifically, we create an Audio-Video CLIP foundation model for Minecraft and an audio prior network which together map audio samples to the latent goal space of the STEVE-1 policy. Additionally, we highlight the tradeoffs that occur when conditioning on different modalities. Our training code, evaluation code, and Audio-Video CLIP foundation model for Minecraft are made open-source to help foster further research into multi-modal generalist sequential decision-making agents.
