Every Bit Counts: A Theoretical Study of Precision-Expressivity Tradeoffs in Quantized Transformers

Quantization reduces the numerical precision of Transformer computations and is widely used to accelerate inference, yet its effect on expressivity remains poorly characterized. We demonstrate a fine-grained theoretical tradeoff between expressivity and precision: For every p we exhibit a function Γ, inspired by the equality function, and prove that a one-layer softmax Transformer can compute Γ, with p bits of precision, but not with p-1 bits of precision. This result concretely explains the widely observed phenomenon of empirical loss of expressivity when quantization is used. Practically, it suggests that tasks requiring equality-like comparisons (exact match, membership, etc.) are especially sensitive to quantization. Dropping even one bit can cross a threshold where the model cannot represent the needed comparison reliably. Thus, it paves the way for developing heuristics that will help practitioners choose how much quantization is possible: the precision should be chosen as a function of the length of equality to be checked for the specific task. Our proofs combine explicit finite-precision Transformer constructions with communication-complexity lower bounds, yielding a tight “one-bit” threshold.

💡 Research Summary

The paper provides a rigorous theoretical analysis of how quantization—reducing the numerical precision of transformer computations—affects the expressive power of transformer models. While prior work has documented empirical accuracy drops when moving from full‑precision to low‑bit formats such as INT8, INT4, or FP8, this study quantifies the exact precision threshold required to compute a fundamental class of functions: equality checks. The authors define a variant of the equality function, TₙEQₘ, which compares two blocks of m bits within an n‑bit input and outputs 1 if they are identical. Their main result, presented informally as Theorem 1.1, states that for any integer p > 0 and any input length n > 4p, there exists a specific instance of TₙEQ_{2p‑1} that cannot be computed by any one‑layer transformer whose arithmetic is limited to p‑1 bits of fixed‑point precision, yet can be computed exactly by a one‑layer transformer using p bits of precision. This “one‑bit” threshold is tight: dropping a single bit of precision eliminates the ability to perform the equality test.

The lower‑bound proof leverages communication‑complexity arguments. By partitioning the input into two independent halves, the authors show that with p‑1 bits the attention scores cannot convey enough information to distinguish the case where the two halves are identical from the case where they differ by a single bit. Consequently, the softmax layer cannot assign a probability of one to the correct token, and the equality function fails. The upper‑bound construction demonstrates how to set attention weights, scaling factors, and biases so that, with exactly p bits, the transformer can generate sufficiently distinct scores to make the softmax output a deterministic 1 for matching inputs and 0 otherwise.

The paper also extends the analysis to floating‑point formats, where numbers are represented by a mantissa and an exponent. The authors prove analogous results for linear transformers and near‑tight results for softmax transformers, showing that the mantissa’s bit‑width plays the same critical role as the total bit‑budget in fixed‑point representations.

Beyond the theoretical contributions, the authors discuss practical implications. Tasks that rely heavily on exact matching—such as token‑level identity checks, database key lookups, password verification, or certain NLP subtasks like semantic equivalence detection—are especially vulnerable to quantization. The results suggest that quantization‑aware training or fine‑tuning cannot fully recover lost expressive power if the underlying hardware limits the precision below the proven threshold. Consequently, practitioners should compute the minimum required precision based on the length of equality checks inherent in their tasks, rather than applying a uniform low‑bit quantization across all components.

In conclusion, the work establishes a precise, bit‑level trade‑off between numerical precision and the class of functions a transformer can represent. It provides a foundation for designing quantization schemes that respect task‑specific expressive requirements, and it opens avenues for future research on multi‑layer transformers, other logical primitives, and empirical validation of the theoretical bounds on modern accelerator hardware.