This educational PyTorch repository implements various distributed training parallelism techniques, including DP, FSDP, TP, and PP, from scratch. It explicitly writes forward/backward logic and collectives, allowing users to directly understand the algorithms and communication patterns without high-level abstractions.
Macrocosmos has introduced ResBM, a new transformer-based architecture for low-bandwidth pipeline-parallel training. It achieves 128x activation compression without significant loss in convergence compared to uncompressed baselines.
Sign-Muon is a novel 1-bit, matrix-aware optimizer proposed for communication-efficient distributed training of large neural networks. It combines sign aggregation with a polar-step framework, achieving an O(1/√T) nonconvex rate and reducing the stochastic term by 1/√M for M workers.
This article differentiates Google's distributed training solutions (DiLoCo/DiPaCo) from the Quadratic Intelligence Swarm (QIS) protocol. It highlights that while Google's tools optimize large-scale training of single models, QIS focuses on decentralized routing of learning outcomes among multiple institutions without centralizing data.