A nearly tight memory-redundancy trade-off for one-pass compression

Let $s$ be a string of length $n$ over an alphabet of constant size $\sigma$ and let $c$ and $\epsilon$ be constants with (1 \geq c \geq 0) and (\epsilon > 0). Using (O (n)) time, (O (n^c)) bits of memory and one pass we can always encode $s$ in (n H_k (s) + O (\sigma^k n^{1 - c + \epsilon})) bits for all integers (k \geq 0) simultaneously. On the other hand, even with unlimited time, using (O (n^c)) bits of memory and one pass we cannot always encode $s$ in (O (n H_k (s) + \sigma^k n^{1 - c - \epsilon})) bits for, e.g., (k = \lceil (c + \epsilon / 2) \log_\sigma n \rceil).

💡 Research Summary

This paper investigates the fundamental trade‑off between memory usage and redundancy for one‑pass compression algorithms operating on data streams. The authors consider a string s of length n over a constant‑size alphabet σ, and they study algorithms that read the input exactly once while using only O(n^c) bits of working memory, where 0 ≤ c ≤ 1 is a constant. Their main result is a nearly tight bound: for any constants c and ε > 0, there exists a one‑pass algorithm that runs in O(n) time, uses O(n^c) bits of memory, and always produces an encoding of length

  n · H_k(s) + O(σ^k · n^{1‑c+ε})

for every order k ≥ 0 simultaneously. Here H_k(s) denotes the k‑th order empirical entropy of s, i.e., the minimum per‑character self‑information achievable when the compressor is allowed to look at contexts of length at most k. The upper bound is achieved by partitioning the input into roughly n^{1‑c+ε/2} blocks, each of size O(n^{c‑ε/2}), compressing each block independently with the Burrows‑Wheeler Transform based algorithm of Grossi, Gupta, and Vitter (which attains n · H_k + O(σ^k log n) per block), and concatenating the results. Super‑additivity of empirical entropy guarantees that the sum of the blockwise entropies does not exceed n · H_k(s), while the additive term contributed by each block accumulates to O(σ^k · n^{1‑c+ε}). The authors also handle the practical case where n is not known in advance by dynamically doubling an estimate of n, which only increases the number of blocks by a logarithmic factor and does not affect the asymptotic redundancy.

To show that this bound cannot be substantially improved, the paper presents a matching lower bound. They fix k = ⌈(c + ε/2)·log_σ n⌉ and consider a randomly chosen σ‑ary de Bruijn sequence d of order k (which contains every possible k‑tuple exactly once). The string s is defined as a repetition of d, possibly truncated at the end. Because every k‑tuple in d is followed by a unique next character, the k‑th order empirical entropy H_k(s) equals zero. Consequently, any encoding that respects the k‑context restriction could, in principle, be as short as O(n · H_k(s)) = 0 bits. However, the Kolmogorov complexity of d is Θ(σ^k) = Θ(n^{c+ε/2}), which exceeds the allowed memory O(n^c). In a one‑pass setting with limited memory, the compressor cannot reconstruct d on the fly; it must output at least Ω(σ^k) bits while processing each copy of d. Since s contains Θ(n / |d|) copies of d, the total output size is Ω(n), which is larger than the claimed upper bound by a factor of n^{ε}. This demonstrates that any one‑pass algorithm using O(n^c) memory cannot guarantee an encoding length of n · H_k(s) + O(σ^k · n^{1‑c‑ε}) for the chosen k.

The paper thus establishes a nearly optimal memory‑redundancy trade‑off for one‑pass compression: the additive redundancy term must be on the order of σ^k · n^{1‑c}, and this term cannot be reduced by more than a sub‑polynomial factor without increasing the memory beyond O(n^c). The result refines our understanding of the limits of streaming compression, showing that while sublinear memory suffices to approach the entropy bound, there is an unavoidable penalty that grows with both the alphabet size and the chosen context length. The techniques combine classic information‑theoretic arguments (entropy super‑additivity, Noiseless Coding Theorem) with combinatorial constructions (de Bruijn sequences) and algorithmic tools (Burrows‑Wheeler Transform), offering a clear and rigorous characterization of what is achievable in memory‑constrained streaming environments.