仮想ロボットの過去行動を要約・質問応答する学習
Learning to Summarize and Answer Questions about a Virtual Robot's Past Actions
自己中心視点の動画から仮想ロボットの行動系列を自然言語で要約し、質問に答える単一モデルを訓練した。
著者: Chad DeChant, Iretiayo Akinola, Daniel Bauer
分類: cs.RO, cs.AI, cs.CL, cs.LG
原文アブストラクト
When robots perform long action sequences, users will want to easily and reliably find out what they have done. We therefore demonstrate the task of learning to summarize and answer questions about a robot agent's past actions using natural language alone. A single system with a large language model at its core is trained to both summarize and answer questions about action sequences given ego-centric video frames of a virtual robot and a question prompt. To enable training of question answering, we develop a method to automatically generate English-language questions and answers about objects, actions, and the temporal order in which actions occurred during episodes of robot action in the virtual environment. Training one model to both summarize and answer questions enables zero-shot transfer of representations of objects learned through question answering to improved action summarization. % involving objects not seen in training to summarize.
