EMO: Pretraining mixture of experts for emergent modularity

EMO proposes a pretraining approach for Mixture of Experts (MoE) models, aiming to achieve emergent modularity. This method focuses on developing specialized components within the model during the pretraining phase.