Husformer:マルチモーダル人間状態認識のためのマルチモーダルトランスフォーマー
Husformer: A Multi-Modal Transformer for Multi-Modal Human State Recognition
複数の生体信号をクロスモーダルトランスフォーマーで融合し、人間の感情や認知負荷を高精度に認識するエンドツーエンド手法を提案した。
著者: Ruiqi Wang, Wonse Jo, Dezhong Zhao, Weizheng Wang, Baijian Yang, Guohua Chen, Byung-Cheol Min
分類: cs.HC, cs.RO
原文アブストラクト
Human state recognition is a critical topic with pervasive and important applications in human-machine systems. Multi-modal fusion, the combination of metrics from multiple data sources, has been shown as a sound method for improving the recognition performance. However, while promising results have been reported by recent multi-modal-based models, they generally fail to leverage the sophisticated fusion strategies that would model sufficient cross-modal interactions when producing the fusion representation; instead, current methods rely on lengthy and inconsistent data preprocessing and feature crafting. To address this limitation, we propose an end-to-end multi-modal transformer framework for multi-modal human state recognition called Husformer. Specifically, we propose to use cross-modal transformers, which inspire one modality to reinforce itself through directly attending to latent relevance revealed in other modalities, to fuse different modalities while ensuring sufficient awareness of the cross-modal interactions introduced. Subsequently, we utilize a self-attention transformer to further prioritize contextual information in the fusion representation. Using two such attention mechanisms enables effective and adaptive adjustments to noise and interruptions in multi-modal signals during the fusion process and in relation to high-level features. Extensive experiments on two human emotion corpora (DEAP and WESAD) and two cognitive workload datasets (MOCAS and CogLoad) demonstrate that in the recognition of human state, our Husformer outperforms both state-of-the-art multi-modal baselines and the use of a single modality by a large margin, especially when dealing with raw multi-modal signals. We also conducted an ablation study to show the benefits of each component in Husformer.
関連論文
- MatchFusion: 時空間マルチモーダル自動運転のための明示的・暗黙的インスタンスマッチングマルチモーダル融合
- LiDAR-BIND-T: ロボティクス応用に向けた時間的一貫性を考慮したセンサモダリティ変換と融合の改良マルチモーダル融合
- t-READi: 自動運転向けTransformer駆動の堅牢で効率的なマルチモーダル推論マルチモーダル融合
- スコアに合わせて演じる:ロボットマニピュレーションのための段階誘導型動的マルチセンサ融合マルチモーダル融合
- 潜在コンタクトを活用したカメラ-LiDAR融合による難条件下の場所認識マルチモーダル融合
- 遮蔽や情報欠損に対応するマルチビュー視覚融合のためのアンサンブル学習:実世界データによる評価と運転者手活動認識への応用マルチモーダル融合
