MPC注入:オフポリシー歩行強化学習を制御器誘発の行動盆地へバイアスする手法
MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins
強化学習が非実用的な局所解に陥る問題を解決するため、モデル予測制御器の遷移をリプレイバッファに注入し、設計者が好む歩容へ学習を誘導する手法を提案した。報酬設計や敵対的模倣学習の複雑さを避けつつ、シミュレーションと実機で効果を実証した。
詳しい要約
1. どんなもの?
2. 先行研究と比べてどこがすごい?
3. 技術・手法の肝は?
4. どうやって有効だと検証した?
5. 議論はある?
6. 次に読むべき論文は?
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Roy Xing, Seyoung Ree, Brian Plancher
分類: cs.RO
原文アブストラクト
Reinforcement learning (RL) for locomotion frequently converges to locally optimal but undeployable behaviors, such as vibrating limbs or scooting on the torso, that maximize return without producing a usable gait. We present MPC-Injection, a low-overhead method that steers RL toward a designer-preferred gait by inserting transitions into the replay buffer from a model predictive controller solving the same Markov decision process. Unlike reward shaping, MPC-Injection does not require redesigning the task reward, and unlike adversarial imitation learning, it adds no discriminator, no kinematic retargeting, and no auxiliary objective. Instead, the controller's preferred behavior is transferred to the policy purely through the replay state distribution. On a 2D walker in simulation and with sim-to-real evaluation on a Go2 quadruped, we show that MPC-Injection drives the policy into the controller's behavior basin using a one to two-term task reward, producing gaits qualitatively comparable to those of reward shaping with twenty-one tuned terms and of adversarial motion priors without their discriminator and retargeting overhead. We further analyze how the injected transitions bias actor-critic updates toward controller-visited states, allowing the policy to learn behaviors that pure RL may fail to reach under simple reward functions.
