文脈構造を活用した有用な補助タスクの生成
Exploiting Contextual Structure to Generate Useful Auxiliary Tasks
強化学習において、与えられたタスクの時相論理表現と大規模言語モデルを用いて文脈に応じた補助タスクを自動生成し、経験の再利用を最大化するマルチタスク強化学習フレームワークを提案した。
著者: Benedict Quartey, Ankit Shah, George Konidaris
分類: cs.AI, cs.LG, cs.RO
原文アブストラクト
Reinforcement learning requires interaction with an environment, which is expensive for robots. This constraint necessitates approaches that work with limited environmental interaction by maximizing the reuse of previous experiences. We propose an approach that maximizes experience reuse while learning to solve a given task by generating and simultaneously learning useful auxiliary tasks. To generate these tasks, we construct an abstract temporal logic representation of the given task and leverage large language models to generate context-aware object embeddings that facilitate object replacements. Counterfactual reasoning and off-policy methods allow us to simultaneously learn these auxiliary tasks while solving the given target task. We combine these insights into a novel framework for multitask reinforcement learning and experimentally show that our generated auxiliary tasks share similar underlying exploration requirements as the given task, thereby maximizing the utility of directed exploration. Our approach allows agents to automatically learn additional useful policies without extra environment interaction.
