日本フィジカルAI新聞

世界のフィジカルAIを、日本語で。

オフライン強化学習arXiv:2210.13846

オフラインからオンライン強化学習への安定した適応のための適応的行動クローニング正則化

Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning

シェア:XThreadsFacebookLINEはてブBluesky

オフライン強化学習で事前学習したエージェントをオンライン微調整する際、行動クローニング損失の重みを性能と安定性に基づいて適応的に調整し、さらにQ関数のランダム化アンサンブルを用いることで、性能崩壊を防ぎつつサンプル効率を向上させる手法を提案。

著者: Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, Joni Pajarinen

分類: cs.LG, cs.RO

原文アブストラクト

Offline reinforcement learning, by learning from a fixed dataset, makes it possible to learn agent behaviors without interacting with the environment. However, depending on the quality of the offline dataset, such pre-trained agents may have limited performance and would further need to be fine-tuned online by interacting with the environment. During online fine-tuning, the performance of the pre-trained agent may collapse quickly due to the sudden distribution shift from offline to online data. While constraints enforced by offline RL methods such as a behaviour cloning loss prevent this to an extent, these constraints also significantly slow down online fine-tuning by forcing the agent to stay close to the behavior policy. We propose to adaptively weigh the behavior cloning loss during online fine-tuning based on the agent's performance and training stability. Moreover, we use a randomized ensemble of Q functions to further increase the sample efficiency of online fine-tuning by performing a large number of learning updates. Experiments show that the proposed method yields state-of-the-art offline-to-online reinforcement learning performance on the popular D4RL benchmark. Code is available: \url{https://github.com/zhaoyi11/adaptive_bc}.

関連論文

  1. 1Boston Dynamics、車輪付きAtlasの研究試作機を紹介研究・技術
  2. 2newmoとKDDIが資本業務提携 2028年に堺市で自動運転タクシー商用化目指す提携・買収
  3. 3GXOとApptronik、Apolloの物流現場試験を拡大 最大1,000台の展開を構想量産・展開
  4. 4NVIDIA、CoRLでGR00T 2とIsaac Lab 3.0を発表新製品・発表
  5. 5UC BerkeleyとNVIDIAの研究者、AIエージェントでロボット制御コードを仮想環境で改善研究・技術

PR・アフィリエイト広告

MINISFORUM(ミニPC)

MINISFORUM(ミニPC)の公式広告バナーMINISFORUM公式ストアを見る ↗
PR本紙発行元 EmplifAI