想像上の遷移を用いたロボット政策学習：ロバスト性と最適性のトレードオフの緩和
Learning Robotic Policy with Imagined Transition: Mitigating the Trade-off between Robustness and Optimality
理想環境で得た最適政策とダイナミクスモデルから想像上の遷移を生成し、それを実演入力として強化学習に組み込む二段階フレームワークを提案。四足歩行の学習において、ドメインランダム化による保守性を抑えつつ、訓練加速・追従誤差低減・分布外ロバスト性向上を実現した。
著者: Wei Xiao, Shangke Lyu, Zhefei Gong, Renjie Wang, Donglin Wang
分類: cs.RO
原文アブストラクト
Existing quadrupedal locomotion learning paradigms usually rely on extensive domain randomization to alleviate the sim2real gap and enhance robustness. It trains policies with a wide range of environment parameters and sensor noises to perform reliably under uncertainty. However, since optimal performance under ideal conditions often conflicts with the need to handle worst-case scenarios, there is a trade-off between optimality and robustness. This trade-off forces the learned policy to prioritize stability in diverse and challenging conditions over efficiency and accuracy in ideal ones, leading to overly conservative behaviors that sacrifice peak performance. In this paper, we propose a two-stage framework that mitigates this trade-off by integrating policy learning with imagined transitions. This framework enhances the conventional reinforcement learning (RL) approach by incorporating imagined transitions as demonstrative inputs. These imagined transitions are derived from an optimal policy and a dynamics model operating within an idealized setting. Our findings indicate that this approach significantly mitigates the domain randomization-induced negative impact of existing RL algorithms. It leads to accelerated training, reduced tracking errors within the distribution, and enhanced robustness outside the distribution.
