Foundation-Preserving Adaptation via Generalized Rayleigh-Quotient Optimization

This paper introduces Foundation Preserving LoRA (FoLoRA), an optimization framework that addresses the degradation of nontarget capabilities during finetuning of foundation models. It uses a generalized Rayleigh quotient to balance task utility and forgetting penalty, guiding updates to preserve pretraining knowledge.