A new book and open-source code are released, detailing how to build modern LLM architectures like GPT-2, Llama 3, and DeepSeek from scratch in PyTorch. It explains the architectural changes required to transform GPT-2 into Llama 3 and implements DeepSeek's advanced features.
Conventional Mixture-of-Experts designs increase parameters linearly with depth by assigning private expert sets to each transformer layer. A new approach, UniPool, replaces this with a single, globally shared pool of experts that all routers draw from, significantly reducing the total expert parameter count while maintaining comparable predictive quality.
Looped Transformers provide a way to improve model performance by iteratively reusing blocks without increasing parameter count, but they suffer from training instability at higher loop iterations. This instability is attributed to gradient oscillation and residual explosion, leading to the proposal of the Fully Looped Transformer, which introduces a Fully Looped Architecture and Attention Injection.
This research introduces a Temporal-Channel Transformer designed for 3D Lidar-based video object detection. It aims to improve the perception capabilities of autonomous driving systems by processing sequential Lidar data.
O artigo investiga a técnica de "noise steering", que injeta perturbações gaussianas em modelos Transformer durante a inferência, para gerar histórias educacionais em árabe. O método melhora a diversidade narrativa para avaliações de leitura de nível inicial, mantendo a qualidade e o nível de leitura.
This work introduces a B-spline-based decoupling framework for compressing Transformer models. It generalizes existing tensor-based methods, addressing their limitations in numerical instability or limited expressiveness by exploiting the properties of B-splines.
This paper investigates the internal mechanisms of knowledge editing methods such as ROME and MEMIT, revealing that diverse edits share a common functional structure reliant on a specific subset of weights. A binary mask over these edited weights reverses most changes by eliminating overattention in later layers, demonstrating this mechanism's necessity for successful edits.