信頼できないシミュレータに注意:強化学習における報酬なしバックドア攻撃
Beware Untrusted Simulators -- Reward-Free Backdoor Attacks in Reinforcement Learning
シミュレータのダイナミクスを改ざんして、報酬を操作せずに強化学習エージェントに行動レベルのバックドアを仕込み、特定のトリガーで危険な行動を引き起こす攻撃手法「Daze」を提案し、実機ロボットへの転移も示した。
詳しい要約
1. どんなもの?
2. 先行研究と比べてどこがすごい?
3. 技術・手法の肝は?
4. どうやって有効だと検証した?
5. 議論はある?
6. 次に読むべき論文は?
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Ethan Rathbun, Wo Wei Lin, Alina Oprea, Christopher Amato
分類: cs.CR, cs.LG, cs.RO
原文アブストラクト
Simulated environments are a key piece in the success of Reinforcement Learning (RL), allowing practitioners and researchers to train decision making agents without running expensive experiments on real hardware. Simulators remain a security blind spot, however, enabling adversarial developers to alter the dynamics of their released simulators for malicious purposes. Therefore, in this work we highlight a novel threat, demonstrating how simulator dynamics can be exploited to stealthily implant action-level backdoors into RL agents. The backdoor then allows an adversary to reliably activate targeted actions in an agent upon observing a predefined ``trigger'', leading to potentially dangerous consequences. Traditional backdoor attacks are limited in their strong threat models, assuming the adversary has near full control over an agent's training pipeline, enabling them to both alter and observe agent's rewards. As these assumptions are infeasible to implement within a simulator, we propose a new attack ``Daze'' which is able to reliably and stealthily implant backdoors into RL agents trained for real world tasks without altering or even observing their rewards. We provide formal proof of Daze's effectiveness in guaranteeing attack success across general RL tasks along with extensive empirical evaluations on both discrete and continuous action space domains. We additionally provide the first example of RL backdoor attacks transferring to real, robotic hardware. These developments motivate further research into securing all components of the RL training pipeline to prevent malicious attacks.
関連論文
- バックドアが部分観測性と出会うとき:実世界強化学習への攻撃強化学習セキュリティ
