An ML team documented the technical challenges faced while fine-tuning and deploying Gemma-4. Key issues included PEFT's incompatibility with Gemma 4's custom layers, SFTTrainer silently breaking KV-sharing attention, and DeepSpeed ZeRO-3 saving half-empty LoRA adapters.
A project successfully added eight Indian languages (Telugu, Kannada, Bengali, Tamil, Malayalam, Marathi, Gujarati, and Hindi) to the Chatterbox-Multilingual TTS model using LoRA adapters and tokenizer extension. This approach trained only 1.4% of the model's parameters, avoiding the complex phoneme engineering typically required for each language.
Um desenvolvedor treinou um modelo Qwen3.5-9B com LoRA para atuar como analista de dados agente, focando em autonomia através de pesos. O modelo alcançou 89% de conclusão de fluxos de trabalho de ponta a ponta sem intervenção humana, superando a falha total do modelo base.
Aletheia introduces a gradient-guided layer selection method for LoRA fine-tuning, identifying the most task-relevant layers and applying adapters selectively with asymmetric rank. This approach achieves a significant 15-28% training speedup across diverse large language models and architectures while broadly matching downstream behavior.
Este artigo investiga a geração de código cross-lingual, focando em métodos de fine-tuning paramétrico-eficiente (PEFT) e otimizadores para LLMs. Os autores demonstram que o fine-tuning LoRA no Code Llama 7B, com um dataset pequeno de alta qualidade, pode superar o desempenho de modelos mais amplamente fine-tuned, e que otimizadores como Sophia oferecem convergência mais rápida com resultados finais comparáveis.
This study systematically applies parameter-efficient fine-tuning (PEFT) using Low-Rank Adaptation (LoRA) to Qwen2.5-3B for a telecommunications customer support conversational assistant. It evaluates 16 LoRA configurations, varying hyperparameters and target modules, using a combinatorial synthetic data generation approach.
This article explains how the LoRA (Low-Rank Adaptation) technique enables fine-tuning billion-parameter language models on consumer hardware like laptops. Instead of updating all parameters, LoRA adds tiny trainable modules, drastically reducing GPU memory requirements.
LoRA (Low-Rank Adaptation) is introduced as the leading PEFT method, enabling efficient adaptation of massive LLMs like Llama 3 without requiring extensive hardware resources. The post promises to delve into LoRA's mathematical intuition, the concept of "intrinsic dimension," and its game-changing impact for AI engineers.
TalkLoRA propõe um framework MoELoRA que aborda a instabilidade de roteamento e a dominância de especialistas em métodos existentes, permitindo a comunicação entre especialistas antes do roteamento. Isso é feito através de um Módulo de Conversação leve, que facilita a troca de informações, gerando um sinal de roteamento mais robusto para Large Language Models (LLMs).
This research discovers that LoRA fine-tuning leads to 'un-learning' on contested examples, where high annotator disagreement correlates with increased loss during training. This pattern is distinct from full fine-tuning and consistently observed across multiple encoder and decoder-only models and datasets.
This paper introduces ReLoRA, a knowledge-reusing re-adaptation framework that efficiently restores service-ready LoRA adapters for evolving LLM services. It addresses the computational cost of retraining and quality degradation from naive application to updated base models.
HELLoRA proposes a novel method for fine-tuning Mixture-of-Experts (MoE) models by applying Low-Rank Adaptation (LoRA) modules only to the most frequently activated experts at each layer. This technique significantly reduces trainable parameters and improves downstream performance, attributing its success to structured regularization that maintains expert specialization.
This research systematically investigates LoRA placement in hybrid language models, which combine attention and recurrent components. It finds that adapting the attention pathway consistently outperforms full-model adaptation with significantly fewer parameters, while the effect of adapting the recurrent backbone varies drastically depending on the hybrid architecture (sequential vs. parallel).
This paper investigates how adaptation methods (Full FT vs. LoRA) and optimization scale jointly shape attention drift and transfer retention in fine-tuned CLIP models. A controlled matched-learning-rate comparison reveals that the learning rate strongly modulates structural change, with Full FT showing marked contraction at higher rates while LoRA remains entropy-positive.
BaLoRA is a Bayesian extension of LoRA that enhances the accuracy of large-scale model adaptation. This novel approach not only quantifies uncertainty but also significantly narrows the performance gap with full fine-tuning.
This paper decomposes an evolutionary Mixture-of-LoRA system, examining factors such as router rewrite, per-domain evaluation, and an adaptation lifecycle. Results indicate that the router rewrite is solely responsible for the balanced log-PPL improvement observed.
This content details the fine-tuning process of the NVIDIA Cosmos Predict 2.5 model. It leverages LoRA/DoRA techniques for robot video generation applications.
This content presents a runnable ComfyUI workflow for rendering AI characters with shop items, combining LoRA for character stability and IP-Adapter for reference image features. It details how to balance these techniques, recommending moderate IP-Adapter weight and early handoff to avoid distorting the character's face.
Sparse Memory Finetuning (SMF) addresses catastrophic forgetting in pretrained language models by updating only a small subset of memory rows. Experiments show SMF improves performance on a medical exam task while substantially mitigating forgetting compared to LoRA and full finetuning.
The author trained an AI model using 22 photos of their cat to enable it to generate images of the pet in various scenes, employing the LoRA technique. This article details the second day of the experiment, focusing on photo preparation and selection criteria to teach the AI the cat's distinctive features.