A senior engineer spent the past year implementing five LLM architectures from scratch in PyTorch, including GPT-2, Llama 3, and DeepSeek. The project resulted in open-source code and a detailed book documenting the process, explaining advanced concepts like KV cache, MoE, and FP8 quantization.
AutoCompress is a transformer compression method based on the empirical finding that Layer 0 carries disproportionately high task-critical information. Its Critical Layer Isolation (CLI) architecture achieves 2.47x compression on GPT-2 Medium with 59.5% parameter reduction, significantly outperforming a uniform bottleneck baseline.
The Cognitive Categorical Transformer (CCT) is a 306M-parameter architecture that augments a pretrained GPT-2 Small backbone with cognitively grounded components derived from category theory and cognitive science inspirations. It achieved a 12% relative reduction in perplexity on WikiText-103 compared to a fine-tuned GPT-2 Small baseline, with 84% of the architectural improvement attributed to GT-Full simplicial message passing.
This content provides a guide for reproducing the GPT-2 (124M) model, detailing the steps required to recreate this language architecture. It serves as a practical tutorial for AI enthusiasts and developers.
