This paper investigates the mechanistic origins of catastrophic forgetting in Large Language Models (LLMs), comparing Reinforcement Learning (RL) with Supervised Fine-Tuning (SFT). It reveals that RL preserves internal computational circuits more effectively, mitigating the forgetting of prior capabilities, unlike SFT which causes greater circuit disruption.
Researchers found that language model performance can significantly improve when deeper layers learn context-free value vectors, preserving original token information. This eliminates the need to recompute or persistently cache these values, as the context-dependent component provides little additional benefit.
This study introduces Gait2Hip-60, a deep learning framework to predict hip muscle forces and joint moments directly from multi-cadence gait kinematics. It compares LSTM, Transformer, and Mamba models, evaluating their performance on healthy adults and an external cohort of patients.
This study evaluates Transformer and LSTM frameworks for streamflow inference in ungauged basins under limited hydrological information. The LSTM architecture showed stronger overall performance than the Transformer model, and incorporating downstream information further boosted performance for all models.
Geometry-Aware Tabular Diffusion (GATD) is introduced to improve tabular synthesis by augmenting denoisers with pairwise angles and lengths computed from column value differences. It achieves state-of-the-art performance with fewer parameters, reducing Shape and Trend error, and showing that explicit relational supervision drives the gains.
This paper introduces Tensor Cache, a two-level cache for Transformers designed to optimize KV caches. It pairs sliding-window softmax attention (L1) with a fixed-size outer-product fast-weight memory (L2) to manage evicted tokens, improving access to relevant evidence outside the context window.
Unicorn is a new framework for scalable, high-dimensional time series forecasting, bridging the gap between channel-independent and channel-dependent models. It leverages a latent prototype codebook to learn universal correlation patterns, significantly outperforming state-of-the-art architectures, especially in few-shot transfer scenarios.
This research introduces FuRA (Full-Rank Adaptation), a novel parameter-efficient fine-tuning method that addresses limitations in existing techniques by incorporating spectral preconditioning. By reparameterizing weight matrices via full-rank Singular Value Decomposition and constraining updates, FuRA outperforms unconstrained Full Fine-Tuning while maintaining efficiency.
This research systematically studies structured state space models (SSMs) for time-series classification, comparing complex Mamba-based architectures with simpler diagonal SSMs (S4D). Surprisingly, S4D consistently outperforms Mamba variants in accuracy and efficiency on large-scale benchmarks, challenging the assumption that increased model complexity leads to better performance in this domain.
Liquid Neural Networks (LNNs) model hidden state evolution as a continuous differential equation, addressing the limitations of discrete-time RNNs and LSTMs in capturing fluid temporal dynamics. This paper benchmarks LNNs against LSTMs across four sequential modalities, revealing LNNs' superior parameter efficiency and robustness, especially in native temporal domains and clinical environments.
The article explains that Transformers reuse the same set of weights for queries, keys, and values across all input words, enabling parallel computation. This reusability makes the self-attention mechanism highly efficient.
Este tutorial aborda a tradução automática neural e os modelos sequência a sequência, explorando seus fundamentos e aplicações essenciais no campo da inteligência artificial.
Este conteúdo aborda LongLive, um sistema para a geração interativa e em tempo real de vídeos longos. A tecnologia foca em produzir sequências de vídeo estendidas de forma dinâmica.
This paper offers a comprehensive survey and benchmark of deep learning models applied to time series data. It systematically reviews various architectures and their performance across different tasks and datasets.
This content is an in-depth review of Deep Learning, exploring its fundamentals and advancements. It offers a comprehensive analysis of the techniques and applications within this field of artificial intelligence.
This content describes research focused on improving reconstruction fidelity by utilizing a novel approach called JumpReLU SparseAutoencoders.
This paper proposes an innovative method to enhance pancreas segmentation in CT and MRI images. It utilizes recurrent neural contextual learning and a direct loss function to optimize accuracy.
The title describes a unified CNN architecture for real-time spatiotemporal action localization. This work focuses on improving efficiency and accuracy in detecting activities within videos.
This article offers a modern perspective on the classical bias-variance tradeoff, re-evaluating its application and relevance in the context of contemporary neural networks. It explores how this fundamental concept manifests and impacts performance in deep learning models.
This research explores the use of deep convolutional networks for the specific task of camera identification. It delves into how these advanced AI models can differentiate between various cameras.