O autor compartilha resultados de otimização de um servidor de inferência com duas GPUs para LLMs, alcançando 198 tok/s com o modelo Qwen3.5-122B NVFP4. O conteúdo detalha a configuração de hardware (2x RTX PRO 6000 Blackwell) e compara o desempenho de diferentes motores e modelos de linguagem.
The article discusses how NUMA memory topology, not just VRAM, is a critical bottleneck for LLM inference on multi-socket servers, causing significant throughput degradation. RustChain's RAM Coffers solves this by detecting NUMA topology and optimizing memory allocation and thread pinning for predictable, enhanced performance.
This paper introduces Group-Query Latent Attention (GQLA), a modification to Multi-head Latent Attention (MLA). GQLA exposes two algebraically equivalent decoding paths, allowing a single set of trained weights to adapt efficiently to different hardware platforms like H100 and H20 without retraining.