模倣学習/ポリシー編集2026/6/1arXiv:2606.29201

行動アンクローニング：推論時ステアリングなしでモードリダイレクションをポリシー重みに蒸留する手法

Behavior Uncloning: Distilling Mode Redirection into Policy Weights without Inference-Time Steering

デモデータに含まれる望ましくない行動モードを抑制するため、一時的なモード分類器からのリダイレクション信号をポリシー重みに蒸留し、推論時オーバーヘッドなしで安全な行動を実現する手法MoREを提案した。

詳しい要約

1. どんなもの？

Behavior-cloned policies が demonstration datasets から複数の behavior modes を学習し、deployment 時に unsafe/undesired な mode を含む問題に対処する手法 MoRE (Mode Redirection) を提案。短い "uncloning" step で policy rollouts を desired behavior modes へ redirect する。

2. 先行研究と比べてどこがすごい？

data curation や inference-time steering は original demonstrations へのアクセスや full retraining、inference-time overhead を要する。MoRE は standalone policy として unwanted modes を zero inference-time overhead で抑制し、8 つの simulated/real-world tasks で平均 deployment SR を 44 percentage points 改善。

3. 技術・手法の肝は？

temporary mode classifier からの redirection signal を policy weights へ distill し、retain loss で desired-mode competence を保持しつつ edit を balance。standalone policy が unwanted modes を inference-time overhead なしで抑制。

4. どうやって有効だと検証した？

8 つの simulated および real-world tasks で評価。original mixed-mode policy 比で平均 deployment SR を 44 percentage points 改善。adaptation/steering baselines 中で最強の SR を示し、filtered-data retraining reference に接近。task competence と inference speed を保持。Diffusion Policy や Pi0.5 VLA など robot policy backbones、多様な task categories、real-world deployments へ generalize。

5. 議論はある？

要旨からは不明。

6. 次に読むべき論文は？

Diffusion Policy、Pi0.5 VLA、filtered-data retraining、inference-time steering、data curation などの関連手法。

※ AIが要旨から生成した要約です。正確性は原文をご確認ください。

著者: Hao Wang, Jiuzhou Lei, Dayou Li, Bangya Liu, Minghui Zheng, Manling Li, Ruohan Zhang, Zhiwen Fan

分類: cs.RO, cs.AI

原文アブストラクト

Behavior-cloned policies often learn multiple behavior modes from demonstration datasets, including modes that are unsafe or otherwise undesired at deployment. For example, a policy trained on diverse handover demonstrations may learn to pass a knife blade-first. Standard remedies such as data curation and inference-time steering either require access to the original demonstrations for full retraining or add substantial inference-time overhead. To address this gap, we propose MoRE(Mode Redirection), which redirects policy rollouts toward desired behavior modes through a short "uncloning" step. Specifically, MoRE distills the redirection signal from a temporary mode classifier into the policy weights to steer behavior. A retain loss balances this edit by preserving desired-mode competence, allowing the standalone policy to suppress unwanted modes with zero inference-time overhead. Across eight simulated and real-world tasks, MoRE improves the average deployment success rate (SR) by 44 percentage points over the original mixed-mode policy. Among all compared adaptation and steering baselines, MoRE achieves the strongest SR and approaches the filtered-data retraining reference, while preserving task competence and inference speed. MoRE also generalizes across robot policy backbones, including Diffusion Policy and the Pi0.5 VLA, diverse task categories, and real-world deployments.

arXivで原文を読む →arXiv概要ページ

← 論文一覧