日本フィジカルAI新聞

世界のフィジカルAIを、日本語で。

マルチモーダル予測arXiv:2109.07561

身体性エージェントのためのマルチセンサリ先読みフレームワーク

A Framework for Multisensory Foresight for Embodied Agents

シェア:XThreadsFacebookLINEはてブBluesky

視覚・触覚・音・力覚の4モダリティを探索行動と組み合わせ、将来の感覚状態を予測する教師なしニューラルネットを提案。ヒューマノイドロボットのデータで非視覚情報が視覚予測精度を向上させることを示した。

著者: Xiaohui Chen, Ramtin Hosseini, Karen Panetta, Jivko Sinapov

分類: cs.RO, cs.CV

原文アブストラクト

Predicting future sensory states is crucial for learning agents such as robots, drones, and autonomous vehicles. In this paper, we couple multiple sensory modalities with exploratory actions and propose a predictive neural network architecture to address this problem. Most existing approaches rely on large, manually annotated datasets, or only use visual data as a single modality. In contrast, the unsupervised method presented here uses multi-modal perceptions for predicting future visual frames. As a result, the proposed model is more comprehensive and can better capture the spatio-temporal dynamics of the environment, leading to more accurate visual frame prediction. The other novelty of our framework is the use of sub-networks dedicated to anticipating future haptic, audio, and tactile signals. The framework was tested and validated with a dataset containing 4 sensory modalities (vision, haptic, audio, and tactile) on a humanoid robot performing 9 behaviors multiple times on a large set of objects. While the visual information is the dominant modality, utilizing the additional non-visual modalities improves the accuracy of predictions.

  1. 1Boston Dynamics、車輪付きAtlasの研究試作機を紹介研究・技術
  2. 2newmoとKDDIが資本業務提携 2028年に堺市で自動運転タクシー商用化目指す提携・買収
  3. 3GXOとApptronik、Apolloの物流現場試験を拡大 最大1,000台の展開を構想量産・展開
  4. 4NVIDIA、CoRLでGR00T 2とIsaac Lab 3.0を発表新製品・発表
  5. 5UC BerkeleyとNVIDIAの研究者、AIエージェントでロボット制御コードを仮想環境で改善研究・技術

PR・アフィリエイト広告

MINISFORUM(ミニPC)

MINISFORUM(ミニPC)の公式広告バナーMINISFORUM公式ストアを見る ↗
PR本紙発行元 EmplifAI