実世界概念の言語接地のための実用的クロスモーダル多様体アライメント
Practical Cross-modal Manifold Alignment for Grounded Language
RGB-D画像と言語記述のトリプレットをサンプリングし、トリプレット損失で言語と視覚の一貫したマルチモーダル埋め込みを学習する手法を提案し、複数の評価指標で既存手法を上回った。
著者: Andre T. Nguyen, Luke E. Richards, Gaoussou Youssouf Kebe, Edward Raff, Kasra Darvish, Frank Ferraro, Cynthia Matuszek
分類: cs.CV, cs.LG, cs.RO, stat.ML
原文アブストラクト
We propose a cross-modality manifold alignment procedure that leverages triplet loss to jointly learn consistent, multi-modal embeddings of language-based concepts of real-world items. Our approach learns these embeddings by sampling triples of anchor, positive, and negative data points from RGB-depth images and their natural language descriptions. We show that our approach can benefit from, but does not require, post-processing steps such as Procrustes analysis, in contrast to some of our baselines which require it for reasonable performance. We demonstrate the effectiveness of our approach on two datasets commonly used to develop robotic-based grounded language learning systems, where our approach outperforms four baselines, including a state-of-the-art approach, across five evaluation metrics.
