O autor compartilha resultados de otimização de um servidor de inferência com duas GPUs para LLMs, alcançando 198 tok/s com o modelo Qwen3.5-122B NVFP4. O conteúdo detalha a configuração de hardware (2x RTX PRO 6000 Blackwell) e compara o desempenho de diferentes motores e modelos de linguagem.
This content suggests that users with 16GB VRAM add an old GPU (6GB+ VRAM) to increase total VRAM, enabling the execution of larger LLM models (~30b) even with a weaker secondary card. It includes a practical configuration example for `llama-server`.
A new DGX Spark owner is seeking advice on configuring it for local LLM inference, planning to use vLLM, PyTorch, and Hugging Face models for a private API backend. They are looking for recommendations on efficient models, tuning tips for vLLM on unified memory systems, and real-world throughput insights.
The author explores using an AMD Alveo V80 FPGA card for LLM inference, aiming to approximate the performance of a dedicated Taalas HC1 chip. After consulting Gemini Pro, potential speeds of 1,400-3,200 tk/s were estimated, and the author seeks community input on this speculative approach.
The article discusses how NUMA memory topology, not just VRAM, is a critical bottleneck for LLM inference on multi-socket servers, causing significant throughput degradation. RustChain's RAM Coffers solves this by detecting NUMA topology and optimizing memory allocation and thread pinning for predictable, enhanced performance.
This article highlights the critical and often underestimated financial impact of AI compute, particularly token usage, when deploying AI agents in production. It emphasizes that token budgets, rather than feature roadmaps, define an agent's true operational limits due to direct costs and overheads like RAG.
This guide addresses the challenges of configuring Laravel Horizon for AI inference workloads in production, where standard queue job defaults fail due to the extended processing times of LLMs. It explains how to prevent silent timeouts and job failures that occur when Horizon's default settings are not adapted for long-running AI tasks.
This post showcases a comprehensive observability solution utilizing Amazon Managed Grafana dashboards. It provides a holistic view of both the quality and quantity of LLMs served on Amazon SageMaker AI inference endpoints.
O artigo argumenta que o consumo de energia é o maior gargalo na inferência de LLMs, mais do que largura de banda ou VRAM, devido a limitações físicas. Isso se deve ao colapso da Lei de Dennard por volta de 2006, que impediu a redução automática do consumo de energia com o encolhimento dos transistores.
Este estudo caracteriza a sobrecarga de despacho do WebGPU para inferência de LLM em diversas plataformas de GPU, backends e navegadores. Ele revela que benchmarks simples superestimam os custos e identifica o verdadeiro custo por despacho da API WebGPU, destacando a necessidade dessa distinção para otimizações eficazes.
AWS is optimizing LLM inference with speculative decoding on Trainium and vLLM, and the Spring AI SDK for Bedrock AgentCore is now generally available. New research also explores agentic system failures, CNN uncertainty quantification, and LLMs' role in clinical reasoning.