Transformers with Selective Access to Early Representations [R]

The paper introduces SATFormer, a new Transformer variant that improves efficiency by allowing heads to selectively re-access early representations instead of uniformly copying them. This context-dependent gating mechanism optimizes the reuse of information, offering a better efficiency-performance trade-off.