VLA/強化学習/巧緻操作2026/5/1arXiv:2605.30226

BORA: オフライン強化学習とオンライン残差適応を橋渡しする実世界巧緻操作VLAモデル

BORA: Bridging Offline Reinforcement Learning and Online Residual Adaptation for Real-World Dexterous VLA Models

実世界の巧緻操作向けに、オフライン学習で価値関数を構築し、オンラインで人間介入による残差適応を行うVLAモデルの後訓練フレームワークを提案した。

詳しい要約

1. どんなもの？

BORAは、実世界のdexterous VLAモデル向けのoffline-to-online RL post-trainingフレームワーク。VLAモデルは視覚言語理解をロボット操作に接地するが、高次元のhand controlと誤差蓄積によりdexterous manipulationは困難。BORAはofflineでcriticを構築し、onlineでVLAを凍結して軽量なHuman-in-the-Loop (HiL) chunk-wise residual adaptationを行う。

2. 先行研究と比べてどこがすごい？

従来のpure imitation learningやtraditional decoupled RLベースラインと比較して、BORAは標準設定で平均成功率33%絶対向上、未見物体汎化で最大43%改善。offline criticとintervention-driven rewardsを継承し、pretrained policyを安定priorとして保持しつつ実世界の物理的ばらつきに適応する点が優位。

3. 技術・手法の肝は？

- offline phase: VLMのcognition tokensとaction chunksを入力とするcriticを構築し、action-conditioned value guidanceを実現。 - online phase: VLA baseを凍結し、Human-in-the-Loop (HiL) chunk-wise residual adaptationを導入。 - offline criticを継承し、intervention-driven rewardsで実行誤差を修正。 - pretrained policyを安定priorとして保持。

4. どうやって有効だと検証した？

5つの複雑な実世界dexterousタスクで評価。pure imitation learningおよびtraditional decoupled RLベースラインと比較し、標準設定で平均成功率33%絶対向上、未見物体汎化で最大43%改善を達成。

5. 議論はある？

高次元dexterous explorationはtemporal inconsistency、sample inefficiency、hardware risksを引き起こす。BORAはこれらに対処するが、要旨からは限界や失敗事例、計算コスト、安全性の詳細は不明。

6. 次に読むべき論文は？

要旨で参照/比較されているpure imitation learning、traditional decoupled RL、VLAモデル、offline-to-online RL、Human-in-the-Loop (HiL) adaptation、residual adaptation、action chunking、critic learningなどの関連研究。具体的な論文名は要旨からは不明。

※ AIが要旨から生成した要約です。正確性は原文をご確認ください。

著者: Zhongxi Chen, Yifan Han, Yanming Shao, Huanming Liu, Congsheng Xu, Xiaoyu Chen, Yao Mu, Wenzhao Lian

分類: cs.RO, cs.AI

原文アブストラクト

Vision-Language-Action (VLA) models have emerged as a promising paradigm for grounding visual-language understanding into real-world robotic manipulation. However, dexterous manipulation remains challenging for VLA policies due to high-dimensional hand control and compounding execution errors, which makes real-world RL post-training essential for bridging the gap between visually grounded action generation and physically reliable dexterous execution. However, high-dimensional dexterous exploration often triggers temporal inconsistency, sample inefficiency and hardware risks in the real world. To address these challenges, we propose BORA, an offline-to-online RL post-training framework designed for real-world dexterous VLA models. In the offline phase, BORA constructs a critic that takes both the VLM's cognition tokens and action chunks as inputs. This design enables action-conditioned value guidance, allowing the critic to evaluate dexterous hand motions beyond visual context alone. During the subsequent online phase, BORA freezes the VLA base and introduces a lightweight, Human-in-the-Loop (HiL) chunk-wise residual adaptation mechanism to mitigate real-world execution errors and further correct the offline-learned intents within the actual physical environment. By inheriting the offline critic and employing intervention-driven rewards, BORA effectively corrects execution discrepancies and adapts to real-world physical variances while preserving the pretrained policy as a stable prior. Extensive evaluations across five complex real-world dexterous tasks demonstrate that BORA significantly outperforms pure imitation learning and traditional decoupled RL baselines, achieving a 33% absolute increase in average success rate under standard settings and up to a 43% improvement in unseen object generalization.

arXivで原文を読む →arXiv概要ページ

← 論文一覧