ViTaPEs: マルチモーダルトランスフォーマーにおけるクロスモーダルアラインメントのための視触覚位置エンコーディング
ViTaPEs: Visuotactile Position Encodings for Cross-Modal Alignment in Multimodal Transformers
視覚と触覚のペア入力からタスク非依存の視触覚表現を学習するトランスフォーマーアーキテクチャViTaPEsを提案し、二段階の位置エンコーディング注入によりクロスモーダルな空間推論を強化、複数の実世界データセットで認識タスクの性能向上とゼロショット汎化を実証した。
詳しい要約
1. どんなもの?
2. 先行研究と比べてどこがすごい?
3. 技術・手法の肝は?
4. どうやって有効だと検証した?
5. 議論はある?
6. 次に読むべき論文は?
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Fotios Lygerakis, Ozan Özdenizci, Elmar Rückert
分類: cs.CV, cs.LG, cs.RO
原文アブストラクト
Tactile sensing provides local essential information that is complementary to visual perception, such as texture, compliance, and force. Despite recent advances in visuotactile representation learning, challenges remain in fusing these modalities and generalizing across tasks and environments without heavy reliance on pre-trained vision-language models. Moreover, existing methods do not study positional encodings, thereby overlooking the multi-stage spatial reasoning needed to capture fine-grained visuotactile correlations. We introduce ViTaPEs, a transformer-based architecture for learning task-agnostic visuotactile representations from paired vision and tactile inputs. Our key idea is a two-stage positional injection: local (modality-specific) positional encodings are added within each stream, and a global positional encoding is added on the joint token sequence immediately before attention, providing a shared positional vocabulary at the stage where cross-modal interaction occurs. We make the positional injection points explicit and conduct controlled ablations that isolate their effect before a token-wise nonlinearity versus immediately before self-attention. Experiments on multiple large-scale real-world datasets show that ViTaPEs not only surpasses state-of-the-art baselines across various recognition tasks but also demonstrates zero-shot generalization to unseen, out-of-domain scenarios. We further demonstrate the transfer-learning strength of \emph{ViTaPEs} in a robotic grasping task, where it outperforms state-of-the-art baselines in predicting grasp success. Project page: https://sites.google.com/view/vitapes
