: Pre-defined sparsity levels (e.g., 1% outliers) to ensure predictable memory usage.
Below is an informative paper-style summary of the technology represented by this identifier.
SpQR represents a shift from uniform quantization to . By treating weights differently based on their importance, it bridges the gap between massive model scales and accessible hardware. SPQR.SPQRAlive.18.var
: Despite the hybrid structure, optimized kernels allow for faster inference compared to uncompressed models due to reduced memory bandwidth bottlenecks. 4. Implementation (SPQRAlive.18.var)
Traditional quantization methods, such as , often struggle with "outlier" weights—individual parameters that have a disproportionate impact on the model's output. When these outliers are forced into low-bit representations (like 4-bit), the model's perplexity (accuracy) degrades significantly. 2. Technical Mechanism : Pre-defined sparsity levels (e
SpQR: Sparse-Quantized Representation for Near-Lossless LLM Compression
: The remaining "non-sensitive" weights are quantized to a low bit-width (e.g., 3 or 4 bits) using a very small group size to minimize local error. By treating weights differently based on their importance,
: It is the first method to allow 3-4 bit quantization with almost no measurable loss in perplexity compared to the 16-bit baseline.