分布強化学習によるリスクを考慮した四足歩行の学習
Learning Risk-Aware Quadrupedal Locomotion using Distributional Reinforcement Learning
価値分布を推定してリスク指標を組み込んだPPO（DPPO）を提案し、四足ロボットANYmalでリスク回避からリスク志向まで単一パラメータで動的に制御できる歩行を実現した。
著者: Lukas Schneider, Jonas Frey, Takahiro Miki, Marco Hutter
分類: cs.RO, cs.LG
原文アブストラクト
Deployment in hazardous environments requires robots to understand the risks associated with their actions and movements to prevent accidents. Despite its importance, these risks are not explicitly modeled by currently deployed locomotion controllers for legged robots. In this work, we propose a risk sensitive locomotion training method employing distributional reinforcement learning to consider safety explicitly. Instead of relying on a value expectation, we estimate the complete value distribution to account for uncertainty in the robot's interaction with the environment. The value distribution is consumed by a risk metric to extract risk sensitive value estimates. These are integrated into Proximal Policy Optimization (PPO) to derive our method, Distributional Proximal Policy Optimization (DPPO). The risk preference, ranging from risk-averse to risk-seeking, can be controlled by a single parameter, which enables to adjust the robot's behavior dynamically. Importantly, our approach removes the need for additional reward function tuning to achieve risk sensitivity. We show emergent risk sensitive locomotion behavior in simulation and on the quadrupedal robot ANYmal. Videos of the experiments and code are available at https://sites.google.com/leggedrobotics.com/risk-aware-locomotion.
