Moonshot AI has open-sourced FlashKDA, a CUTLASS C++ kernel for Kimi Delta Attention, offering up to 2.22x performance improvement over the Triton baseline on H20 benchmarks. This new implementation integrates with flash-linear-attention and enhances linear attention architectures like KDA.
This paper introduces Elastic Attention Cores as a new building block for scalable Vision Transformers, addressing the high cost of dense self-attention. The approach uses a core-periphery block-sparse attention structure and nested dropout for elastic inference cost adjustments, achieving competitive accuracy.
![Elastic Attention Cores for Scalable Vision Transformers [R]](/cdn-cgi/image/width=3840,quality=75,format=webp/https://preview.redd.it/zjea47ez7w0h1.png?width=140&height=140&crop=1:1,smart&auto=webp&s=2017a3d330a172670baae5645ddff3137bbe1df6)
This article discusses the architectural novelties of DeepSeek V4, highlighting its hybrid attention system (CSA + HCA) and Manifold-Constrained Hyper-Connections. It also touches on frontier-scale FP4 QAT training, differentiating it from previous models.
This paper investigates the dispatch-overhead bottleneck that prevents token pruning from fully realizing latency reductions in Vision Transformers (ViTs). It proposes a lightweight Triton attention kernel with a lower dispatch floor, achieving up to 2.24x end-to-end throughput for pruned ViTs.
UniMamba is a new unified spatial-temporal forecasting framework that integrates efficient state-space dynamics with attention-based dependency learning to tackle multivariate time series challenges. It employs a Mamba Variate-Channel Encoding Layer and a Spatial Temporal Attention Layer to capture both global temporal dependencies and inter-variate correlations.
This work introduces a method for image caption generation, utilizing region-based attention and scene factorization to enhance descriptive relevance and accuracy. It aims to more effectively align visual perception with textual narration.
This research systematically evaluates variants of the Query, Key, and Value (QKV) attention formulation in Transformers, including shared key-value, query-key, and single projections. Experiments across synthetic, vision, and language modeling tasks demonstrate that these alternative formulations perform on par or occasionally better than standard QKV Transformers, with Q-K=V sharing offering significant KV cache reduction in language modeling.
This research investigates LLM prompt sensitivity by comparing instruction-based and example-based prompting styles. It finds that despite performance variation, LLMs share common underlying mechanisms, specifically "lexical task heads" which are attention heads that literally describe the task and trigger answer production.
This research unifies Transformers, diffusion-maps, and magnetic Laplacians, presenting them as different regimes of a single Markov geometry built from pre-softmax query-scores. It defines a QK "bidivergence" to connect attention and diffusion, organizing their dynamics with product of experts and Schrödinger-bridges.
Este artigo propõe LPC-SM, uma arquitetura híbrida autorregressiva para modelos de linguagem de contexto longo, que separa atenção local, memória persistente, correção preditiva e controle em tempo de execução. O modelo de 158M parâmetros é avaliado, demonstrando melhorias na perda de LM e estabilidade em sequências longas.
Super Apriel, a 15B-parameter supernet, has been released, offering four trained mixer choices per decoder layer to enable multiple speed/quality presets from a single checkpoint. This allows for 2.9x to 10.7x decode throughput gains with 96% to 77% quality retention, and also facilitates speculative decoding without a separate draft model.
LACE is a novel framework enabling Large Language Models (LLMs) to coordinate and share insights across multiple parallel reasoning paths through cross-thread attention. It leverages a synthetic data pipeline to teach collaborative error-correction, leading to over 7 points improvement in reasoning accuracy.
This paper introduces ANDRE, a novel Attention-based Neuro-symbolic Differentiable Rule Extractor (ILP) framework for learning first-order logic programs. It optimizes over a continuous rule space with fully differentiable, attention-driven logical operators, addressing scalability challenges in noisy and probabilistic settings.
This research presents a novel robust invisible video watermarking method that leverages attention mechanisms to enhance imperceptibility and resilience against attacks.
This article delves into the limitations of LSTMs in maintaining context, even with their improved memory capabilities over vanilla RNNs. The author uses a personal experience of learning English to illustrate the three specific problems LSTMs still don't solve, setting the stage for discussing attention mechanisms.
Targeted, tiny weight edits to specific attention heads in LLMs, as demonstrated by the ASGuard method, can drastically reduce jailbreak success rates from linguistic tricks. This surgical approach patches vulnerabilities by dampening activations in relevant attention heads, maintaining overall model competence while significantly enhancing safety.
This research introduces a novel approach to neural language correction leveraging character-based attention mechanisms. The method aims to improve the accuracy and robustness of automatically correcting grammatical and spelling errors in text.
This research systematically investigates LoRA placement in hybrid language models, which combine attention and recurrent components. It finds that adapting the attention pathway consistently outperforms full-model adaptation with significantly fewer parameters, while the effect of adapting the recurrent backbone varies drastically depending on the hybrid architecture (sequential vs. parallel).
LayerBoost proposes an optimization for LLMs by selectively modifying the attention mechanism based on the sensitivity of individual transformer layers. This aims to reduce the quadratic complexity of softmax attention, a major bottleneck for efficient inference, without significant model quality degradation.
This paper introduces Gist Sparse Attention (GSA), an end-to-end learnable method to scale large language models to long contexts without architectural modifications. GSA compresses context into 'gist tokens' for summary, then selectively restores relevant raw chunks for detailed attention, combining compact global representations with targeted fine-grained access.