Prism ML announced Ternary Bonsai, a new family of 1.58-bit language models designed to balance strict memory constraints with high accuracy. These models, available in 8B, 4B, and 1.7B sizes, achieve a 9x smaller memory footprint than 16-bit models while outperforming most peers.
This content discusses the process of AI model distillation, focusing on how to reduce massive models with over 100 billion parameters to significantly smaller versions with less than 4 billion. The aim is to enhance the efficiency and accessibility of complex AI models.
This paper proposes a unified compressed-sensing-guided framework for dynamic LLM execution, addressing the massive parameter counts, memory use, and decoding latency of large language models. It integrates model and prompt compression by using random measurement operators and sparse recovery to estimate task-conditioned and token-adaptive support sets.
LiftQuant is a novel framework for continuous bit-width control in Large Language Models, addressing limitations of integer-based quantization. It employs a "lift-then-project" mechanism to achieve quasi-continuous bit-width tuning for optimal deployment.
ReAD proposes a Reinforcement-guided Capability Distillation framework for Large Language Models, aiming to compress LLMs while preserving essential abilities for downstream tasks. It explicitly accounts for the interdependence of capabilities, optimizing token budget usage and mitigating degradation of useful abilities.
The article compares traditional quantization methods (like INT4/INT8) used for local LLMs with the emerging 1.58-bit ternary quantization approach found in projects like BitNet b1.58. It highlights the simplicity of ternary models, which use only -1, 0, or +1 for weights, contrasting them with standard post-training quantization techniques.
This study investigates the impact of post-training quantization on Large Language Models (LLMs) quality, revealing that compression can lead to bias emergence. 3-bit quantization caused 6-21% of previously unbiased items to develop new stereotypical behaviors in models like Qwen2.5-7B, Mistral-7B, and Phi-3.5-mini. This follows a clear dose-response pattern across various precision levels.
ModelBest has released BitCPM-CANN, the first 1.58-bit ternary LLM trained end-to-end on Ascend 910B NPUs. This model uses 6x less VRAM than BF16 while retaining most capabilities and is available in four open-source sizes.