Temporal Structure

EMA Is Not All You Need: Mapping the Boundary Between Structure and Content in Recurrent Context

This research explores Exponential Moving Average (EMA) traces as a minimal recurrent context to delineate the capabilities and limitations of fixed-coefficient accumulation in sequence models. It demonstrates that EMA traces excel at encoding temporal structure, matching advanced models on structural tasks, yet fundamentally fail to capture token identity, resulting in significantly reduced performance for language modeling.