Actor-Critic

Why dynamically routing multi-timescale advantages in PPO causes policy collapse (and a simple decoupled fix) [R]

An undergrad AI researcher discovered why fusing multi-timescale advantages in PPO Actor-Critic architectures leads to policy collapse. This occurs due to surrogate objective hacking and the router's preference for short-term horizons because of lower temporal uncertainty.