A user is struggling to integrate Chebyshev filters into a CNN architecture to improve performance, noting that current results are similar to baseline. They are seeking advice on filter integration, placement, tuning, and whether others have found benefits.
Moonshot AI has open-sourced FlashKDA, a CUTLASS C++ kernel for Kimi Delta Attention, offering up to 2.22x performance improvement over the Triton baseline on H20 benchmarks. This new implementation integrates with flash-linear-attention and enhances linear attention architectures like KDA.
A new PyTorch optimizer named 'Rose' has been released, promising low VRAM usage, fast convergence, and excellent generalization, licensed under Apache 2.0. Developed over several years, it aims to be easy to use and more memory-efficient than 8-bit AdamW.
This content suggests that users with 16GB VRAM add an old GPU (6GB+ VRAM) to increase total VRAM, enabling the execution of larger LLM models (~30b) even with a weaker secondary card. It includes a practical configuration example for `llama-server`.
This content discusses the perspective that Large Language Models (LLMs) learn in a reverse manner and that the scalability hypothesis has inherent limits.
A scientist with dual industry and academic affiliation shares insights on developing a fundamental scientific theory of machine learning, based on approximately seven years of work. The post outlines thoughts on achieving a true science of deep learning.
This paper introduces Elastic Attention Cores as a new building block for scalable Vision Transformers, addressing the high cost of dense self-attention. The approach uses a core-periphery block-sparse attention structure and nested dropout for elastic inference cost adjustments, achieving competitive accuracy.
![Elastic Attention Cores for Scalable Vision Transformers [R]](/cdn-cgi/image/width=3840,quality=75,format=webp/https://preview.redd.it/zjea47ez7w0h1.png?width=140&height=140&crop=1:1,smart&auto=webp&s=2017a3d330a172670baae5645ddff3137bbe1df6)
This content refers to a research paper titled "Follow the Mean: Reference-Guided Flow Matching". It explores a new methodology in generative models.
![Follow the Mean: Reference-Guided Flow Matching [R]](/cdn-cgi/image/width=3840,quality=75,format=webp/https://preview.redd.it/5pleq5b4861h1.png?width=140&height=91&auto=webp&s=5f80ce290c30e51700f9b9fd0f907ee56e9382b2)
A mechanical engineering graduate with a software development background is deciding between two professional master's paths: pursuing a minor in computer science to reapply to MILA for ML/DL, or accepting an offer from Polytechnique Montréal. The decision weighs a longer academic path to strengthen theoretical foundations against starting professional experience sooner.
This discussion questions whether production Vision-Language Models (VLMs) still rely on fixed-patch Vision Transformers (ViTs) for their vision capabilities, despite the existence of more efficient tokenization methods. It explores potential reasons for this, such as marginal gains, pipeline limitations, or unclear scaling laws for adaptive patching.
The paper introduces SATFormer, a new Transformer variant that improves efficiency by allowing heads to selectively re-access early representations instead of uniformly copying them. This context-dependent gating mechanism optimizes the reuse of information, offering a better efficiency-performance trade-off.
![Transformers with Selective Access to Early Representations [R]](/cdn-cgi/image/width=3840,quality=75,format=webp/https://preview.redd.it/bfj0qllk9fzg1.png?width=140&height=47&auto=webp&s=afd139021e7256d039453286e5a71d859d7fe9bb)
The content details a persistent problem with achieving low accuracy (~50%) using self-supervised learning methods like BYOL, MAE, and VICReg for hyperspectral crop stress detection. Despite trying various techniques, performance remains barely better than random for three classes, leading to suspicions about data separability or SSL method suitability.
AutoMuon, a new Python package, enables the Muon optimizer to be used as a drop-in replacement for AdamW in PyTorch training pipelines. It automatically identifies and applies the appropriate optimizer for each model parameter, using Muon for weight matrices and AdamW for other components.
This article, part of the "LLM Neuroanatomy" series, posits that Large Language Models primarily process information geometrically rather than through linguistic representations. It explores the internal mechanisms and structural organization of these advanced AI models.
This research introduces HALO-Loss, a novel method for training neural networks to abstain from making predictions when uncertain. It allows models to express "I don't know" rather than providing potentially incorrect answers, improving reliability.
!["I don't know!": Teaching neural networks to abstain with the HALO-Loss. [R]](/cdn-cgi/image/width=3840,quality=75,format=webp/https://preview.redd.it/loxsfywek4vg1.png?width=140&height=123&auto=webp&s=e494bc934351aeddad8b50b76059d35dc7987a4f)
This article discusses the architectural novelties of DeepSeek V4, highlighting its hybrid attention system (CSA + HCA) and Manifold-Constrained Hyper-Connections. It also touches on frontier-scale FP4 QAT training, differentiating it from previous models.
A user observed that INT8 quantization in their deep learning model yielded better inference accuracy than FP16, which was unexpected. They are seeking explanations for INT8's superior performance over FP16.
A Deep Learning practitioner is seeking resources such as papers, courses, and tutorials to learn about AI for Materials Science. The goal is to gain sufficient knowledge to conduct meaningful research in the area and contribute to the community, with a UChicago course already identified as a benchmark.
In this video, the author explains the pipeline for training a neural network model using Convolutional Neural Networks (CNN) for landmark image classification. Those interested can follow the training pipeline and test the model on the author's Github.
This article, part of a PyTorch series, details the neural network training process by demonstrating a nested loop structure to iterate through training data. It explains how to calculate total loss, derive output, and apply the loss function for model optimization using `loss.backward()`.