Exploring the Limits of Pruning: Task-Specific Neurons, Model Collapse, and Recovery in Task-Specific Large Language Models

This study explores the existence of task-specific neurons in large language models, focusing on mathematical reasoning and code generation. It introduces an activation-based selectivity metric for neuron pruning, which consistently outperforms random pruning in reducing computational cost and preserving task accuracy, while preventing performance collapse.