Most people starting with local LLMs jump straight to 4-bit quantization because it's fast and uses

This article compares 16-bit, 8-bit, and 4-bit LLM quantization, revealing that 4-bit, while faster, significantly compromises quality on reasoning and math tasks. The real trade-off is between the task and required precision, with 8-bit being optimal for precision-demanding tasks, offering minimal quality loss with only a slight speed reduction. Quantization choice should be based on the task and hardware considerations, not solely on hardware.