PEFT

7 items

RESEARCHarXiv CS.LG·4/9/2026

FLeX: Fourier-based Low-rank EXpansion for multilingual transfer

Este artigo investiga a geração de código cross-lingual, focando em métodos de fine-tuning paramétrico-eficiente (PEFT) e otimizadores para LLMs. Os autores demonstram que o fine-tuning LoRA no Code Llama 7B, com um dataset pequeno de alta qualidade, pode superar o desempenho de modelos mais amplamente fine-tuned, e que otimizadores como Sophia oferecem convergência mais rápida com resultados finais comparáveis.

Cross-lingual code generation PEFT LoRA LLM Fine-tuning

RESEARCHarXiv CS.CL·5d ago

PEFT of SLM for Telecommunications Customer Support: A Comparative Study of LoRA Configurations with Energy Consumption Analysis

This study systematically applies parameter-efficient fine-tuning (PEFT) using Low-Rank Adaptation (LoRA) to Qwen2.5-3B for a telecommunications customer support conversational assistant. It evaluates 16 LoRA configurations, varying hyperparameters and target modules, using a combinatorial synthetic data generation approach.

Telecommunications LLMs customer support PEFT

ARTICLEDEV.to AI·4/22/2026

Why LoRA? Understanding the representative PEFT

LoRA (Low-Rank Adaptation) is introduced as the leading PEFT method, enabling efficient adaptation of massive LLMs like Llama 3 without requiring extensive hardware resources. The post promises to delve into LoRA's mathematical intuition, the concept of "intrinsic dimension," and its game-changing impact for AI engineers.

LLMs deep learning fine-tuning PEFT

RESEARCHarXiv CS.LG·4/28/2026

Parameter Efficiency Is Not Memory Efficiency: Rethinking Fine-Tuning for On-Device LLM Adaptation

This research challenges the assumption that Parameter-Efficient Fine-Tuning (PEFT) equates to memory efficiency for on-device LLMs, showing existing methods can still lead to out-of-memory errors. It introduces LARS (Low-memory Activation-Rank Subspace), a novel framework that decouples memory consumption from sequence length by constraining the activation subspace, achieving an average 33.54% memory footprint reduction.

Memory Optimization on-device AI fine-tuning PEFT

RESEARCHarXiv CS.LG·4/6/2026

LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning

O LiME (Lightweight Mixture of Experts) propõe uma nova abordagem para MoE-PEFT, utilizando modulação leve de um único módulo PEFT compartilhado em vez de adaptadores separados por especialista. Isso reduz significativamente os parâmetros, introduz roteamento de parâmetros zero e generaliza para qualquer método PEFT, superando as limitações de escalabilidade e aplicabilidade.

multi-task learning model efficiency Deep Learning Architectures Mixture of Experts

RESEARCHarXiv CS.CL·26d ago

PEML: Parameter-efficient Multi-Task Learning with Optimized Continuous Prompts

This paper introduces PEML, a method for parameter-efficient multi-task learning with optimized continuous prompts for Large Language Models. It addresses the shortcomings of existing PEFT methods like LoRA and Prefix Tuning by enabling more efficient fine-tuning across multiple tasks and facilitating resource consolidation.

Resource efficiency multi-task learning LLMs Prompt tuning

RESEARCHarXiv CS.LG·4/6/2026

An Initial Exploration of Contrastive Prompt Tuning to Generate Energy-Efficient Code

Este estudo explora a otimização de LLMs para gerar código mais energeticamente eficiente, utilizando Contrastive Prompt Tuning (CPT). A CPT, que combina Contrastive Learning e Prompt Tuning, é avaliada em Python, Java e C++ para promover o desenvolvimento de software mais verde.

LLMs Energy Efficiency code generation PEFT