The author implemented and open-sourced reproductions of two recent ideas, Cartridges and STILL, for neural KV-cache compaction and long-context inference. The goal is to make these research ideas easy to inspect and run with benchmark code, also comparing them against existing methods.
O texto explora uma abordagem experimental para indexação de vetores HNSW que utiliza embeddings quantizados de 3 bits, em vez de float32, para reduzir o uso de memória. A técnica, baseada em PolarQuant, permite cálculo de distância eficiente via tabelas pré-computadas, resultando em economia de memória e bom recall, apesar de um processo de construção mais lento e desafios com o ruído de quantização.
This work addresses the significant memory footprint of Key-Value (KV) caching in transformer language models, proposing optimization through the depth dimension. It introduces a method for cross-layer cache sharing, demonstrating that dropping a layer's cache can be efficient without information loss, and suggests a training approach with random cross-layer attention.
This paper introduces RateQuant, a method for optimal mixed-precision KV cache quantization in large language models to address memory bottlenecks. It tackles the challenge of distortion model mismatch, where applying one quantizer's distortion model to another degrades performance compared to uniform quantization.
MegaTrain é um sistema focado em memória que permite o treinamento eficiente de modelos de linguagem grandes com mais de 100 bilhões de parâmetros em precisão total em uma única GPU. Ele armazena parâmetros na memória do host e utiliza otimizações como um motor de execução pipeline e templates de camada sem estado para superar gargalos de largura de banda e maximizar a utilização da GPU.
LoSA introduces Locality Aware Sparse Attention to address memory-bound attention and the KV Inflation problem in block-wise diffusion language models, especially for long contexts. It optimizes performance by reusing cached attention for stable tokens and applying sparse attention only to active tokens, significantly reducing KV index loading.
DeepSeek V4's true innovation lies in its memory architecture, not just its scale, making its 1M token context practically usable. Through KV cache compression techniques like CSA and HCA, it achieves a nearly 9x memory reduction compared to its predecessor, overcoming practical challenges of long-context models.
The author describes how a 'direction' channel in their Snake AI's state representation caused a 6.4x memory overhead. Using uint8 for only 2 bits of information prevented efficient bit-packing, leading to 1,600 bytes per state instead of 250.
This research challenges the assumption that Parameter-Efficient Fine-Tuning (PEFT) equates to memory efficiency for on-device LLMs, showing existing methods can still lead to out-of-memory errors. It introduces LARS (Low-memory Activation-Rank Subspace), a novel framework that decouples memory consumption from sequence length by constraining the activation subspace, achieving an average 33.54% memory footprint reduction.
This paper introduces BASIS, an efficient backpropagation algorithm designed to mitigate the O(L * BN) spatial memory bottleneck in deep neural networks. It fully decouples activation memory from batch and sequence dimensions, preserving exact error signals while computing weight updates with massively compressed tensors, and addresses gradient instability with novel mechanisms.
This paper introduces LKV (Learned KV Eviction), a novel approach to optimize Key-Value (KV) cache memory in Large Language Models (LLMs). LKV formulates KV compression as an end-to-end differentiable optimization problem, learning budgets and token selection to overcome limitations of heuristic methods.
This work rethinks KV cache eviction for LLMs using an information-theoretic objective derived from the Information Bottleneck principle. It introduces CapKV, a new capacity-aware method that preserves information, outperforming existing heuristic strategies.