Sparse Memory Finetuning

Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning

Sparse Memory Finetuning (SMF) addresses catastrophic forgetting in pretrained language models by updating only a small subset of memory rows. Experiments show SMF improves performance on a medical exam task while substantially mitigating forgetting compared to LoRA and full finetuning.