RMTL: VLM報酬を用いた長時間操作のための強化マイクロタスク学習
RMTL: Reinforced Micro-task Learning for Long-Horizon Manipulation with VLM Rewards
長時間の操作タスクを複数のマイクロタスクに分解し、VLM報酬と逆カリキュラムで学習する階層的強化学習手法を提案。
詳しい要約
1. どんなもの?
2. 先行研究と比べてどこがすごい?
3. 技術・手法の肝は?
4. どうやって有効だと検証した?
5. 議論はある?
6. 次に読むべき論文は?
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Anıl Can Ateş, Orhan Kahraman, Cihan Topal
分類: cs.RO
原文アブストラクト
Reinforcement learning (RL) for robotic manipulation often requires manually designing a dense reward function, which is difficult to tune and often fragile, or learning a reward from human demonstrations or preferences, which can be expensive. A recent line of work uses pretrained vision-language models (VLMs) as zero-shot reward models, replacing these costs with a single text prompt. However, we argue that a single global prompt is too coarse for long-horizon manipulation tasks with randomized initial conditions. The single-prompt VLM reward is near-flat for much of the trajectory, making early progress hard for the agent to detect. We propose Reinforced Micro-Task Learning (RMTL), an approach that decomposes a manipulation task into a small set of language-described micro-tasks and trains the agent to switch between them. At each step, the agent receives a multi-view VLM reward computed using the prompt of the currently active micro-task and averaged across multiple camera views to reduce the effect of view-specific occlusions. A reverse curriculum gradually exposes the agent to harder initial conditions, while a PPO worker is first trained with a fixed distance-based rule that selects the active micro-task. We then replace this rule with a learned hierarchical manager, turning rule-based phase selection into a fully learned hierarchical policy. We instantiate RMTL on the Fetch manipulation environment using three short stage-specific prompts and without additional prompt tuning. Experiments show that RMTL provides more informative reward signals than single-prompt VLM rewards, enabling faster learning. These results suggest that decomposing VLM rewards into micro-task-specific language prompts can substantially improve the scalability of language-guided reinforcement learning for robotic manipulation.
関連論文
- 環境に寄りかかって遠くの物体に手を伸ばすヒューマノイド制御マニピュレーション
- 目的関数をポリシーとして:接触を伴うマニピュレーションのための視覚言語モデルによるコストプログラムマニピュレーション
- 見えない開始遅延が自己対戦ハンドオーバーを未知のパートナーに頑健にするマニピュレーション
- 非線形マニピュレータの単一線形化軌道計画における最適動作点選択マニピュレーション
- VRヒューマンデモで点群把持ポリシーの持続的失敗を修正マニピュレーション
- 低出力二重振り子の自然な振り上げ制御マニピュレーション
