日本フィジカルAI新聞

世界のフィジカルAIを、日本語で。

マルチモーダル動作生成arXiv:2204.03051

知覚・表現・生成:マルチモーダル情報をロボット動作軌道へ変換する枠組み

Perceive, Represent, Generate: Translating Multimodal Information to Robotic Motion Trajectories

シェア:XThreadsFacebookLINEはてブBluesky

視覚や音声などのマルチモーダル入力と指示列から、ロボットが実行する動作軌道列を生成する3段階フレームワークPRGを提案し、文字を書く手書きタスクで有効性を示した。

著者: Fábio Vital, Miguel Vasco, Alberto Sardinha, Francisco Melo

分類: cs.RO, cs.AI, cs.LG

原文アブストラクト

We present Perceive-Represent-Generate (PRG), a novel three-stage framework that maps perceptual information of different modalities (e.g., visual or sound), corresponding to a sequence of instructions, to an adequate sequence of movements to be executed by a robot. In the first stage, we perceive and pre-process the given inputs, isolating individual commands from the complete instruction provided by a human user. In the second stage we encode the individual commands into a multimodal latent space, employing a deep generative model. Finally, in the third stage we convert the multimodal latent values into individual trajectories and combine them into a single dynamic movement primitive, allowing its execution in a robotic platform. We evaluate our pipeline in the context of a novel robotic handwriting task, where the robot receives as input a word through different perceptual modalities (e.g., image, sound), and generates the corresponding motion trajectory to write it, creating coherent and readable handwritten words.

  1. 1Boston Dynamics、車輪付きAtlasの研究試作機を紹介研究・技術
  2. 2newmoとKDDIが資本業務提携 2028年に堺市で自動運転タクシー商用化目指す提携・買収
  3. 3NVIDIA、CoRLでGR00T 2とIsaac Lab 3.0を発表新製品・発表
  4. 4GXOとApptronik、Apolloの物流現場試験を拡大 最大1,000台の展開を構想量産・展開
  5. 5UC BerkeleyとNVIDIAの研究者、AIエージェントでロボット制御コードを仮想環境で改善研究・技術

PR・アフィリエイト広告

FRONTIER(GPU搭載BTOパソコン)

FRONTIER(GPU搭載BTOパソコン)の公式広告バナーFRONTIERでGPU搭載PCを見る ↗
PR本紙発行元 EmplifAI