This research investigates the 'grokking' phenomenon in transformers, finding that the long delay to generalization in arithmetic models stems from a decoder bottleneck. The encoder acquires relevant structural knowledge early, but the decoder struggles to access it, a hypothesis supported by causal interventions like transplanting encoders.
This paper identifies normalized spectral entropy as a scalar order parameter for the grokking transition, where models generalize long after memorization. The research shows that entropy collapse precedes generalization, and causal interventions confirm its critical role, providing a predictive model for grokking onset.
This paper proposes the Inter/Intra-class Distance Ratio (ILDR) as a novel geometric signal for the early detection of 'grokking' in neural networks. ILDR, computed from second-to-last layer representations, indicates geometric reorganization in representation space before validation accuracy improves, outperforming existing detection methods.
This empirical study investigates Tian's (2025) feature repulsion theorem in two-layer network grokking, testing its mechanisms and spectral signatures. It observes a clear structure-mechanism dissociation, with the predicted sign rule robustly holding for similar feature pairs despite a strong activation dependence in the spectral signature.