身体性を伴うマルチモーダル・マルチタスク学習
Embodied Multimodal Multitask Learning
言語条件付き視覚ナビゲーションと身体性質問応答を同時に学習し、単語と視覚対象の対応知識をタスク間で転移するDual-Attentionモデルを提案した。
著者: Devendra Singh Chaplot, Lisa Lee, Ruslan Salakhutdinov, Devi Parikh, Dhruv Batra
分類: cs.LG, cs.AI, cs.CL, cs.RO, stat.ML
原文アブストラクト
Recent efforts on training visual navigation agents conditioned on language using deep reinforcement learning have been successful in learning policies for different multimodal tasks, such as semantic goal navigation and embodied question answering. In this paper, we propose a multitask model capable of jointly learning these multimodal tasks, and transferring knowledge of words and their grounding in visual objects across the tasks. The proposed model uses a novel Dual-Attention unit to disentangle the knowledge of words in the textual representations and visual concepts in the visual representations, and align them with each other. This disentangled task-invariant alignment of representations facilitates grounding and knowledge transfer across both tasks. We show that the proposed model outperforms a range of baselines on both tasks in simulated 3D environments. We also show that this disentanglement of representations makes our model modular, interpretable, and allows for transfer to instructions containing new words by leveraging object detectors.
