シミュレーションに錨を下ろした学習によるオンザフライ適応
Sim-Anchored Learning for On-the-Fly Adaptation
シミュレーションで訓練したRLエージェントを実世界データで微調整する際、シミュレーションの批評家を「設計意図の錨」として活用し、優先行動を保ちながら適応を可能にする手法を提案。レーシングクアッドロータで最大50%の消費電力削減を実現。
著者: Bassel El Mabsout, Shahin Roozkhosh, Siddharth Mysore, Kate Saenko, Renato Mancuso
分類: cs.RO, cs.LG
原文アブストラクト
Fine-tuning simulation-trained RL agents with real-world data often degrades crucial behaviors due to limited or skewed data distributions. We argue that designer priorities exist not just in reward functions, but also in simulation design choices like task selection and state initialization. When adapting to real-world data, agents can experience catastrophic forgetting in important but underrepresented scenarios. We propose framing live-adaptation as a multi-objective optimization problem, where policy objectives must be satisfied both in simulation and reality. Our approach leverages critics from simulation as "anchors for design intent" (anchor critics). By jointly optimizing policies against both anchor critics and critics trained on real-world experience, our method enables adaptation while preserving prioritized behaviors from simulation. Evaluations demonstrate robust behavior retention in sim-to-sim benchmarks and a sim-to-real scenario with a racing quadrotor, allowing for power consumption reductions of up to 50% without control loss. We also contribute SwaNNFlight, an open-source firmware for enabling live adaptation on similar robotic platforms.
