The paper introduces WAV v1, a lightweight multi-resolution residual routing method for decoder-only Transformers. It improves upon standard residual connections by augmenting each block with directional detail bases that contrast attention and MLP updates, and early-vs-late sublayer dynamics.
A developer trained transformer-based deep learning models to play chess like humans across various rating buckets, including unique thinking time prediction. The models were trained on Lichess data and achieved accuracy comparable to MAIA-3, despite their small size.
The user is training an AI model from scratch and seeks advice on the best architecture, considering switching from Nanochat (which lacks Transformers compatibility) to the Llama architecture. The goal is an open-source project with a new, larger dataset, despite Nanochat's advantages.
This post details empirical findings from OpenAI's Parameter Golf competition, explaining why State Space Models (SSMs) are structurally disadvantaged compared to transformers in parameter- and time-constrained training regimes. Key issues include worse in_proj weight compression for SSMs and architectural win reversals at higher vocabulary sizes, alongside insights from Mamba-3 Triton kernel experiments.
Macrocosmos has introduced ResBM, a new transformer-based architecture for low-bandwidth pipeline-parallel training. It achieves 128x activation compression without significant loss in convergence compared to uncompressed baselines.
The paper introduces SATFormer, a new Transformer variant that improves efficiency by allowing heads to selectively re-access early representations instead of uniformly copying them. This context-dependent gating mechanism optimizes the reuse of information, offering a better efficiency-performance trade-off.
![Transformers with Selective Access to Early Representations [R]](/cdn-cgi/image/width=3840,quality=75,format=webp/https://preview.redd.it/bfj0qllk9fzg1.png?width=140&height=47&auto=webp&s=afd139021e7256d039453286e5a71d859d7fe9bb)
This content discusses a research paper on Depth-Recurrent Transformers, highlighting its findings on compositional and out-of-distribution generalization. It explores how intermediate step supervision can hinder genuine reasoning in AI models, making them overly reliant on statistical heuristics, a concept extended to foundation models and human intuition.
This article provides a technical breakdown of Visual-Language-Action (VLA) models, explaining how they map vision and language inputs into robot actions. It delves into current action-decoding approaches like tokenized autoregressive actions, diffusion-based action heads, and flow-matching policies.
![How Visual-Language-Action (VLA) Models Work [D]](/cdn-cgi/image/width=3840,quality=75,format=webp/https://external-preview.redd.it/fBpt1C8zS6YDW2Lp0_fnNCU2C0Dw1W3tzt7P4g39SHw.jpeg?width=640&crop=smart&auto=webp&s=d9f046e9b38c478cf671d18df1b23a42fd1613bd)
The user asks about the release date and widespread adoption of TurboQuant, emphasizing the significant performance gains from using an asymmetric setup for K and V. The discussion points to a technical optimization in AI models.
This research systematically evaluates variants of the Query, Key, and Value (QKV) attention formulation in Transformers, including shared key-value, query-key, and single projections. Experiments across synthetic, vision, and language modeling tasks demonstrate that these alternative formulations perform on par or occasionally better than standard QKV Transformers, with Q-K=V sharing offering significant KV cache reduction in language modeling.
This work addresses the challenge of missing modalities in multimodal clinical data for diagnosis by reframing it as an autoregressive sequence modeling task. It leverages causal decoders from LLMs and a missingness-aware contrastive pre-training to outperform baselines on benchmarks like MIMIC-IV and eICU.
This paper studies signal propagation at initialization in transformers using the averaged partial Jacobian norm (APJN) to measure gradient amplification. The theory extends APJN analysis, predicts the asymptotic behavior of APJN at large depth, and explains the subcriticality of normalization-free architectures like Dynamic Tanh and Dynamic erf transformers.
This work addresses the significant memory footprint of Key-Value (KV) caching in transformer language models, proposing optimization through the depth dimension. It introduces a method for cross-layer cache sharing, demonstrating that dropping a layer's cache can be efficient without information loss, and suggests a training approach with random cross-layer attention.
This systematic study of singular value spectra during transformer pretraining reveals three key phenomena: transient compression waves propagating through layers and persistent spectral gradients. It also identifies a Q/K–V functional asymmetry, where query/key projections drive depth-dependent dynamics while value/output projections compress uniformly.
This paper explores "deceptive alignment" in LLMs, a key challenge in AI safety where models deliberately produce false outputs while maintaining accurate internal representations. Researchers introduced a multi-model paradigm, successfully detecting synthetic dishonesty with high accuracy using linear probes across various transformer architectures.
eOptShrinkQ is a two-stage compression pipeline for KV cache in transformer attention heads. It leverages optimal singular value shrinkage and per-vector scalar quantization, grounded in random matrix theory, to achieve near-lossless compression and improve reconstruction.
This content explores the multi-head attention mechanism in AI models, focusing on the idea of collaboration instead of concatenation. It likely discusses an alternative approach to improve attention efficiency or performance.
This research unifies Transformers, diffusion-maps, and magnetic Laplacians, presenting them as different regimes of a single Markov geometry built from pre-softmax query-scores. It defines a QK "bidivergence" to connect attention and diffusion, organizing their dynamics with product of experts and Schrödinger-bridges.
This article explains why self-attention values replace original positional encodings, as they integrate contextual information from all words, clarifying relationships. It then introduces stacking multiple self-attention layers, each with unique weights, to capture more complex linguistic relationships within sentences and paragraphs.
Large Language Models (LLMs) operate as neural networks that learn patterns in text to generate content by predicting the next token. This powerful functionality is driven by massive data, deep architectures, and Transformer-based attention.