運動学的ドメインランダム化と適応による方策転送
Policy Transfer via Kinematic Domain Randomization and Adaptation
シミュレーションでリンク長などの運動学パラメータをランダム化する方が動的パラメータのランダム化より転送性能が高いことを見出し、仮想運動学パラメータで条件付けした方策群をベイズ最適化で適応させる手法を提案した。
著者: Ioannis Exarchos, Yifeng Jiang, Wenhao Yu, C. Karen Liu
分類: cs.RO, cs.LG, cs.SY, eess.SY
原文アブストラクト
Transferring reinforcement learning policies trained in physics simulation to the real hardware remains a challenge, known as the "sim-to-real" gap. Domain randomization is a simple yet effective technique to address dynamics discrepancies across source and target domains, but its success generally depends on heuristics and trial-and-error. In this work we investigate the impact of randomized parameter selection on policy transferability across different types of domain discrepancies. Contrary to common practice in which kinematic parameters are carefully measured while dynamic parameters are randomized, we found that virtually randomizing kinematic parameters (e.g., link lengths) during training in simulation generally outperforms dynamic randomization. Based on this finding, we introduce a new domain adaptation algorithm that utilizes simulated kinematic parameters variation. Our algorithm, Multi-Policy Bayesian Optimization, trains an ensemble of universal policies conditioned on virtual kinematic parameters and efficiently adapts to the target environment using a limited number of target domain rollouts. We showcase our findings on a simulated quadruped robot in five different target environments covering different aspects of domain discrepancies.
