ResBM: a new transformer-based architecture for low-bandwidth pipeline-parallel training, achieving 128× activation compression [R]

Macrocosmos has introduced ResBM, a new transformer-based architecture for low-bandwidth pipeline-parallel training. It achieves 128x activation compression without significant loss in convergence compared to uncompressed baselines.