optimizers

3 items

RESEARCHarXiv CS.LG·4/9/2026

FLeX: Fourier-based Low-rank EXpansion for multilingual transfer

Este artigo investiga a geração de código cross-lingual, focando em métodos de fine-tuning paramétrico-eficiente (PEFT) e otimizadores para LLMs. Os autores demonstram que o fine-tuning LoRA no Code Llama 7B, com um dataset pequeno de alta qualidade, pode superar o desempenho de modelos mais amplamente fine-tuned, e que otimizadores como Sophia oferecem convergência mais rápida com resultados finais comparáveis.

Cross-lingual code generation PEFT LoRA LLM Fine-tuning

RESEARCHarXiv CS.LG·5/7/2026

A Self-Attentive Meta-Optimizer with Group-Adaptive Learning Rates and Weight Decay

MetaAdamW is a novel optimizer that employs a self-attention mechanism to dynamically adjust per-group learning rates and weight decay, addressing the limitation of uniform hyperparameters in adaptive optimizers. Its attention module is trained via a meta-learning objective, integrating gradient alignment, loss decrease, and generalization gap.

Meta-Learning deep learning learning AI Research

ARTICLEDEV.to AI·4/22/2026

Blog 2: Momentum-Based Optimizers

This blog content discusses momentum-based optimizers, exploring their function and importance in accelerating the training of machine learning models. It details how these algorithms improve the convergence and efficiency of neural networks.

deep learning machine learning AI Algorithms