部分的微分可能報酬に対する改良型時間逆伝播
ABPT: Amended Backpropagation through Time with Partially Differentiable Rewards
部分的にしか微分できない報酬でも、Q値関数を活用して勾配バイアスを軽減し、ドローンの制御方策を効率的に学習する手法を提案。
著者: Fanxing Li, Fangyu Sun, Tianbao Zhang, Danping Zou
分類: cs.RO, cs.AI, cs.LG
原文アブストラクト
Quadrotor control policies can be trained with high performance using the exact gradients of the rewards to directly optimize policy parameters via backpropagation-through-time (BPTT). However, designing a fully differentiable reward architecture is often challenging. Partially differentiable rewards will result in biased gradient propagation that degrades training performance. To overcome this limitation, we propose Amended Backpropagation-through-Time (ABPT), a novel approach that mitigates gradient bias while preserving the training efficiency of BPTT. ABPT combines 0-step and N-step returns, effectively reducing the bias by leveraging value gradients from the learned Q-value function. Additionally, it adopts entropy regularization and state initialization mechanisms to encourage exploration during training. We evaluate ABPT on four representative quadrotor flight tasks \li{in both real world and simulation}. Experimental results demonstrate that ABPT converges significantly faster and achieves higher ultimate rewards than existing learning algorithms, particularly in tasks involving partially differentiable rewards. The code will be released at http://github.com/Fanxing-LI/ABPT.
