The paper introduces WAV v1, a lightweight multi-resolution residual routing method for decoder-only Transformers. It improves upon standard residual connections by augmenting each block with directional detail bases that contrast attention and MLP updates, and early-vs-late sublayer dynamics.
A user expresses confusion regarding how a 27B dense model could outperform a 397B Mixture-of-Experts (MoE) model, specifically mentioning Qwen, and questions the utility of the additional experts.
The content announces the addition of Mimo v2.5 model support in llama.cpp and describes its architecture. MiMo v2.5 is a Sparse MoE model with 310B total and 15B activated parameters, supporting text, image, video, and audio modalities with an impressive context length.
The author questions whether Geometric Deep Learning, by explicitly building symmetries and invariances into its architecture, could significantly reduce or eliminate the need for extensive, data-intensive pre-training. This raises the question of whether current massive-scale pre-training is largely a consequence of architectures lacking inherent invariance.
The author questions Qwen's decision to use the Mixture-of-Experts (MoE) architecture for its coding models, instead of more accurate dense models. They speculate the choice might be related to inference speed and regret the absence of a 14B successor.
Expert Upcycling proposes a method to progressively expand Mixture-of-Experts (MoE) capacity in large language models during continued pre-training. It increases the number of experts via duplication and router extension to provide a warm initialization, aiming to reduce training costs while preserving per-token inference cost.
The Qwen3.6-27B dense model outperformed the Qwen3.6-35B-A3B MoE model on SWE-bench, scoring 77.2% versus 73.4%. This indicates that dense models may be proving more effective for real-world software engineering tasks.
DeepSeek V4 delivers a 1 million-token context that is actually usable, solving the GPU memory issue with a hybrid attention architecture that compresses the KV cache by nearly 9x. This makes it a practical solution for long-context inference, unlike many other models.
This paper introduces the Length Value Model (LenVM), a novel token-level framework for modeling the remaining generation length in autoregressive models. By formulating length modeling as a value estimation problem, LenVM provides an annotation-free, scalable, and effective signal for LLMs and VLMs, improving performance on exact length matching tasks.
The Bicameral Model couples two frozen, pretrained language models via a trainable neural interface on their intermediate hidden states, allowing them to operate in lockstep. This method enables a primary model to drive a task while an auxiliary model uses tools or solves constraints, significantly improving accuracy on tasks like arithmetic and logic puzzles.
SAM 3 focuses on building a unified model architecture for detection and tracking tasks. It aims to improve efficiency and accuracy in computer vision applications.
This article delves into the two-model design powering SAM 3D, an AI initiative from Meta. It explains the architectural choices and engineering rationale behind this AI system.
The topology rewrite for an AI system or model is a fundamental reformulation, not just a bug fix. Further details on this development will be shared as the build matures.