FP8

A First Comprehensive Study of TurboQuant: Accuracy and Performance

A comprehensive study on TurboQuant compares its variants (k8v4, 4bit-nc, k3v4-nc, 3bit-nc) with FP8 for KV-cache quantization. FP8 is recommended as the default, offering 2x capacity with negligible accuracy loss and good performance. TurboQuant variants show limited advantages or significant degradation in accuracy and performance, with 4bit-nc being an option for memory-constrained scenarios.