The paper introduces "bicache", the first KV caching technique for shared prefixes in diffusion language models (DLMs), addressing challenges where existing LLM caching methods fail due to DLMs' bidirectional attention. This new approach aims to unlock high-throughput DLM serving by leveraging observations about shared prefix KVs stability in shallow layers.
Um bug de desempenho foi identificado no cuBLAS para operações de multiplicação de matrizes em GPUs NVIDIA RTX, como a 5090, utilizando apenas 40% da capacidade. O autor demonstrou um kernel customizado que supera o cuBLAS em até 70%, sugerindo otimização deficiente para essas GPUs em comparação com modelos Pro e H-series.
A comprehensive study on TurboQuant compares its variants (k8v4, 4bit-nc, k3v4-nc, 3bit-nc) with FP8 for KV-cache quantization. FP8 is recommended as the default, offering 2x capacity with negligible accuracy loss and good performance. TurboQuant variants show limited advantages or significant degradation in accuracy and performance, with 4bit-nc being an option for memory-constrained scenarios.
Moonshot AI has open-sourced FlashKDA, a CUTLASS C++ kernel for Kimi Delta Attention, offering up to 2.22x performance improvement over the Triton baseline on H20 benchmarks. This new implementation integrates with flash-linear-attention and enhances linear attention architectures like KDA.
This post details empirical findings from OpenAI's Parameter Golf competition, explaining why State Space Models (SSMs) are structurally disadvantaged compared to transformers in parameter- and time-constrained training regimes. Key issues include worse in_proj weight compression for SSMs and architectural win reversals at higher vocabulary sizes, alongside insights from Mamba-3 Triton kernel experiments.
The author investigates why a specific Qwen3.6 27B INT8 Autoround quantization recipe outperforms others, observing the model "thinks" less but provides better outputs in benchmarks. They then replicated this performance with a new GGUF quant, noting both consistently achieve answers faster than UD Q8 K XL.
TurboOCR achieves 270–1200 img/s OCR by optimizing PaddleOCR with C++/CUDA, FP16 TensorRT, fused kernels, and batched processing, addressing the performance bottlenecks of VLM-based approaches. This solution drastically improves throughput for large-scale document processing and is suitable for real-time RAG applications.
This paper investigates the dispatch-overhead bottleneck that prevents token pruning from fully realizing latency reductions in Vision Transformers (ViTs). It proposes a lightweight Triton attention kernel with a lower dispatch floor, achieving up to 2.24x end-to-end throughput for pruned ViTs.
This content demonstrates how AI pair programming, using Claude, can lead to significant application performance gains by efficiently identifying and fixing bottlenecks. It presents real before-and-after results, showing how AI detected complex N+1 queries that humans overlooked.
This hands-on guide focuses on performance tuning in Databricks SQL, detailing how to diagnose and fix bottlenecks. It teaches methods like reducing data scans, optimizing joins, and leveraging caching to make queries faster and cheaper, thereby avoiding common mistakes that lead to high latency and wasted resources.
This content focuses on how to troubleshoot latency issues and optimize the performance of Amazon Bedrock Agents. It offers a practical guide to enhance the efficiency and responsiveness of AI agents.
Modelos de linguagem de difusão discreta (dLLMs) aceleram a geração de texto, mas a decodificação paralela degrada a qualidade ao desconsiderar a dependência entre tokens. DEMASK propõe um preditor leve que estima influências condicionais para guiar o desmascaramento simultâneo, comprovadamente melhorando a qualidade. A técnica resulta em um ganho de velocidade de 1.7 a 2.2x, mantendo ou superando o desempenho.
This post demonstrates how to utilize the SOCI index on publicly available Deep Learning AMIs and Containers to reduce cold start times. It covers the various SOCI modes and provides guidance on efficiently implementing this tool in current workloads.
This article highlights that single-threaded customer service bots are slow due to sequential LLM calls, causing up to 12 seconds latency. It proposes a parallel sub-agent pattern with LangGraph and LangSmith to execute research tasks concurrently, significantly reducing response times to around 6.5 seconds.
The "Parallelization Trap" describes how increasing concurrent AI agents can paradoxically reduce overall system throughput due to coordination and coherence problems. This happens as agents contend for shared context, leading to stale or conflicting information.
O conteúdo aborda a 'Join Bomb', um problema de desempenho em motores de recomendação construídos com Neo4j, causado por travessias bidirecionais não otimizadas em grafos densos. É proposta uma solução comparando uma query Cypher 'ingênua' com uma query otimizada baseada em APOC para eliminar esse gargalo.
This article discusses the challenges of integrating AI features into mobile apps without sacrificing performance, such as speed and battery life. It emphasizes that AI performance in mobile applications is a multifaceted problem involving product, architecture, API, and user experience.
Super Apriel, a 15B-parameter supernet, has been released, offering four trained mixer choices per decoder layer to enable multiple speed/quality presets from a single checkpoint. This allows for 2.9x to 10.7x decode throughput gains with 96% to 77% quality retention, and also facilitates speculative decoding without a separate draft model.
This guide addresses common performance and stability pitfalls in Django development, such as inefficient queries and blocking operations. It highlights how AI assistants, specifically Cursor and Claude Code, can significantly aid in building more robust and efficient Django applications.
This content provides practical advice and lessons learned from Zartek on integrating AI into real mobile and web applications, emphasizing problem-first approaches, performance optimization, reliability, cost-saving through caching, and robust observability. It highlights common pitfalls and effective AI features.