要旨で参照されている関連研究は明示されていないが、VAEベースのロボット学習とPPOの組み合わせに関する研究が関連する。具体的には、VAEを用いた表現学習と強化学習の統合に関する論文や、PPOの分散低減手法に関する論文が挙げられる。また、確率的潜在変数を扱う他のポリシー最適化手法（例えば、SVGDやStein変分勾配法を用いた手法）も関連する。次に読むべき論文としては、VAEとPPOを組み合わせた既存の研究（例："Learning Latent Dynamics for Planning from Pixels"や"Variational Information Maximization for Deep Reinforcement Learning"）や、PPOの改良版（例："Trust Region Policy Optimization"）が考えられるが、要旨からは特定の論文名は不明である。
Variational Autoencoders are widely used to encode high-dimensional and noisy observations in robotics. However, their stochastic latent creates a mismatch with Proximal Policy Optimization (PPO): an effective policy marginalizes over the latent distribution, whereas former implementations estimate its probability ratio and KL divergence using only one latent sample. We identify a fundamental but overlooked theoretical cause: naive single-sample approximations in stochastic latent space induce significant variance and bias in the surrogate loss. To address this, we introduce P^3 (Probabilistic Policy Propagation), a distribution-aware optimization framework for VAE-based policies. $P^3$ couples moment-based probabilistic method for stable and efficient learning with sampling-based calibration for robust policy behavior under latent uncertainty. In our experiments, P^3 boosts data efficiency from 64.6% to >96%, reduces convergence steps by >20%. Furthermore, P^3 is evaluated on challenging humanoid parkour tasks and shows an effective foundation for VAE-based PPO. Code is available at https://github.com/ylyem9x/P3_Open.
