This paper introduces the On-Policy Diffusion Language Model (OPDLM) for transforming autoregressive models (ARLMs) into diffusion language models (DLMs). It addresses issues like knowledge loss and train-inference mismatch by employing On-Policy Distillation (OPD).
Prism ML announced Ternary Bonsai, a new family of 1.58-bit language models designed to balance strict memory constraints with high accuracy. These models, available in 8B, 4B, and 1.7B sizes, achieve a 9x smaller memory footprint than 16-bit models while outperforming most peers.
SenseNova U1 is a new series of native multimodal models that unifies multimodal understanding, reasoning, and generation within a monolithic architecture. These innovative models natively think and act across language and vision, marking a fundamental paradigm shift in multimodal AI.
The author built a diffusion language model from scratch to better understand complex concepts, without the help of AI-generated code. They trained the 7.5M parameter model on the tiny Shakespeare dataset and shared the code on GitHub.
Failures in language model reasoning emerge through distinct processes that leave identifiable token-level signatures. These failures are characterized as "committed failure" or "persistent uncertainty", and understanding these signatures helps distinguish failing from successful completions across various configurations.
This paper proposes a novel technique, Token-to-Mask (T2M) remasking, to refine masked diffusion language models like LLaDA2.1. The method addresses the shortcomings of Token-to-Token (T2T) editing by resetting suspect tokens to a mask state, enabling more accurate re-prediction.
This paper introduces AERIC, a novel transfer-oriented hidden-state approach for anticipatory same-pass monitoring of implicit harmful dialogue in language models. It aims to detect potential risks early enough to prevent the exposure of harmful continuations.
This content explores three post-hoc methods—Platt Scaling, Isotonic Regression, and Temperature Scaling—designed to enhance the calibration of language models. These techniques aim to reduce the disparity between a model's predicted confidence and its actual accuracy.
TALM (Tool Augmented Language Models) focuses on integrating external tools with large language models to augment their capabilities. This approach allows LLMs to perform complex tasks more effectively by leveraging specialized functions and real-world interactions.
This research introduces generic triple-latent sequence models, which use a running token state and compressed pair-memory to capture higher-order token interactions. These models show improvement over a Transformer baseline on language-model benchmarks, though a retrieval extension enhances recall but is slower.
This paper introduces a hybrid pre-training objective for text encoders, combining a JEPA-style latent-space prediction loss with a standard Masked Language Modelling (MLM) objective. This new approach aims to encourage representations anchored to deeper semantic structure rather than just surface-form token identity, showing significantly more uniform embeddings.
This research investigates whether real-data scaling laws are governed by a progressive coverage of a latent predictive contribution spectrum, rather than solely by token-frequency. Using a suffix-automaton and a global-KL predictive contribution spectrum, the study finds a strong correlation between the spectrum's tail slope and the data-scaling exponent of GPT learners, showing that effective truncation rank scales logarithmically.
This paper reveals a critical vulnerability in diffusion-based language models (dLLMs) where their safety alignment, based on monotonic denoising schedules, can be easily bypassed. By re-masking refusal tokens and injecting an affirmative prefix, researchers achieved high attack success rates against prominent dLLMs, exposing a structural flaw.
This paper introduces Learn-by-Wire Guard (LBW-Guard), a bounded autonomous training-control governance layer for language models. It aims to improve training stability and efficiency by observing telemetry and applying bounded control, significantly reducing final perplexity.
GRISP is a novel SPARQL-based question-answering method over knowledge graphs, leveraging a fine-tuned small language model (SLM). It generates SPARQL query skeletons from natural language questions and iteratively refines them by selecting knowledge graph items, achieving state-of-the-art results on Wikidata and Freebase benchmarks.
This research introduces a "finite-answer preference stabilization" method to determine when a language model's answer preference becomes stable before its final output. It shows that this stabilization often occurs before the answer is parseable, with a significant lead time.
This paper presents a comprehensive analysis of neural activation patterns across six distinct large language model (LLM) architectures, examining their performance on twelve cognitive task categories. The findings reveal fundamental differences in how encoder and decoder architectures process diverse cognitive tasks, with mathematical reasoning consistently producing the highest attention entropy and decoder models exhibiting significantly higher sparsity.
This research study reveals that small instruction-tuned language models (LMs) using Chain-of-Thought (CoT) for arithmetic often employ a positional shortcut, copying whichever number occupies the trailing position before the answer delimiter. This shortcut dominates, even when intermediate reasoning is correct, significantly impacting answer accuracy.
This study investigates the effect of discourse-role labels, such as "Reference" or "Instruction," on language model behavior. It reveals that the adoption rate of misleading information can shift significantly (56-84 percentage points) depending on the label, with labels like "Instruction" increasing adoption and "Example" consistently suppressing it.
The article discusses the process of creating a BERT model for the Swedish language, a project developed at the National Library of Sweden. The aim is to enhance natural language processing for Swedish.