A new book and open-source code are released, detailing how to build modern LLM architectures like GPT-2, Llama 3, and DeepSeek from scratch in PyTorch. It explains the architectural changes required to transform GPT-2 into Llama 3 and implements DeepSeek's advanced features.
An updated PyTorch repository features educational implementations of FlashAttention versions FA1 through FA4. The focus is on demonstrating the algorithmic differences and evolution of the method, facilitating an understanding of its design ideas without delving into hardware specifics.
A senior engineer spent the past year implementing five LLM architectures from scratch in PyTorch, including GPT-2, Llama 3, and DeepSeek. The project resulted in open-source code and a detailed book documenting the process, explaining advanced concepts like KV cache, MoE, and FP8 quantization.
LIDARLearn is a unified PyTorch library for 3D point cloud deep learning, supporting 56 ready-to-use configurations and built-in cross-validation. It also automates publication-ready LaTeX PDF generation after training, making it ideal for researchers in 3D computer vision and remote sensing.
![We’re proud to open-source LIDARLearn [R] [D] [P]](/cdn-cgi/image/width=3840,quality=75,format=webp/https://preview.redd.it/53o0rt8wfxvg1.png?width=640&crop=smart&auto=webp&s=17dbca74f07ea72d1d32060979cecfdc47ab51fe)
This educational PyTorch repository implements various distributed training parallelism techniques, including DP, FSDP, TP, and PP, from scratch. It explicitly writes forward/backward logic and collectives, allowing users to directly understand the algorithms and communication patterns without high-level abstractions.
A new PyTorch optimizer named 'Rose' has been released, promising low VRAM usage, fast convergence, and excellent generalization, licensed under Apache 2.0. Developed over several years, it aims to be easy to use and more memory-efficient than 8-bit AdamW.
AutoMuon, a new Python package, enables the Muon optimizer to be used as a drop-in replacement for AdamW in PyTorch training pipelines. It automatically identifies and applies the appropriate optimizer for each model parameter, using Muon for weight matrices and AdamW for other components.
A new DGX Spark owner is seeking advice on configuring it for local LLM inference, planning to use vLLM, PyTorch, and Hugging Face models for a private API backend. They are looking for recommendations on efficient models, tuning tips for vLLM on unified memory systems, and real-world throughput insights.
This article, part of a PyTorch series, details the neural network training process by demonstrating a nested loop structure to iterate through training data. It explains how to calculate total loss, derive output, and apply the loss function for model optimization using `loss.backward()`.
A principal notícia de IA desta semana é a adesão do Safetensors à PyTorch Foundation, um passo crucial para padronizar a infraestrutura. Safetensors, originalmente da Hugging Face, resolve problemas de segurança e eficiência da serialização de modelos PyTorch, substituindo o formato pickle por um método mais seguro e rápido.
This content describes porting Andrej Karpathy's NanoChat from PyTorch to JAX/Flax NNX, achieving fast training on a single GPU and TPU compatibility. It details XLA's advantages in eliminating Python overhead while highlighting its limitations regarding advanced features and debugging.
This content delves into Meta's AI initiatives, covering LLaMA, PyTorch, FAIR, and the new Muse Spark model which significantly boosted its stock. It also touches on Meta AI's NFL draft predictions and its deepening chip collaboration with Broadcom.
Geoopt is a PyTorch library that facilitates Riemannian optimization, offering advanced geometric optimization techniques for machine learning models.
DeepRobust is a PyTorch library designed for research and development in adversarial attacks and defenses. It provides tools for testing the robustness of deep learning models against malicious manipulations.
This article, part of a series, details how to initialize weights and biases within a PyTorch neural network class. It demonstrates using "nn.Parameter" to define a weight, enabling PyTorch to treat it as an optimizable network component.
This article is a beginner's guide to using `torch.profiler` for performance analysis in PyTorch. It explains how to effectively profile deep learning models to identify bottlenecks and optimize execution.
This content covers the implementation of one of the earliest neural image caption generator models using the PyTorch library.
O título anuncia que Safetensors está se juntando à Fundação PyTorch, marcando um desenvolvimento importante no ecossistema de IA.