A Layer-wise Analysis of Supervised Fine-Tuning

This research analyzes Supervised Fine-Tuning (SFT), revealing that instruction-following capabilities emerge distinctly across layers: middle layers are stable while final layers are highly sensitive. Leveraging this, the authors propose Mid-Block Efficient Tuning, which updates critical intermediate layers, outperforming standard LoRA with reduced parameter overhead.