Empirical entropy in context

We trace the history of empirical entropy, touching briefly on its relation to Markov processes, normal numbers, Shannon entropy, the Chomsky hierarchy, Kolmogorov complexity, Ziv-Lempel compression, de Bruijn sequences and stochastic complexity.

💡 Research Summary

The paper provides a sweeping historical and technical survey of empirical entropy, tracing its roots from early 20th‑century probability theory to modern data‑compression and indexing structures. It begins with Markov’s 1906 work on finite‑state stochastic processes, noting his analysis of Pushkin’s text that demonstrated dependence on a short context. The narrative then moves to Borel’s definition of normal numbers, the construction of the first explicit normal number by Champernowne, and the later discovery of absolutely normal numbers, linking these concepts to Turing’s investigations of universal machines and computability.

Shannon’s 1948 information‑theoretic framework is presented next. By axiomatizing entropy, Shannon derived the noiseless coding theorem, establishing that the expected length of an optimal prefix code for a stationary ergodic source lies within one bit of the source’s entropy. He also fitted Markov models of increasing order to English, observing that higher‑order models better approximate natural language, though he apparently ignored the existence of normal numbers such as Champernowne’s.

The paper then contrasts this engineering perspective with Chomsky’s linguistic critique. Chomsky argued that finite‑state Markov models cannot capture syntactic phenomena such as long‑distance agreement or the grammaticality of novel sentences, proposing instead a hierarchy of grammars (regular, context‑free, context‑sensitive, unrestricted). Despite this, the engineering community has largely retained the Markov assumption because many algorithmic problems become tractable under it, while unrestricted grammars correspond to Turing‑equivalent systems and lead to incomputable optimization problems (e.g., the smallest grammar problem).

Kolmogorov complexity is introduced as a language‑independent measure of the information content of an individual string, defined as the length of the shortest program that outputs the string. The paper notes the incomputability of Kolmogorov complexity and the large additive constants that arise from changing description languages. Lempel and Ziv’s 1976 work on a computable complexity metric based on the maximal number of distinct non‑overlapping substrings is described; this metric motivated the classic LZ77 and LZ78 compression schemes. Their analysis showed that LZ78’s compression ratio is asymptotically bounded below any finite‑state transducer, regardless of the source, while later work by Cover and Thomas assumed stationary ergodic sources for LZ analysis despite acknowledging that natural language may not satisfy these assumptions.

Empirical entropy, introduced by Kosaraju and Manzini in 1999, is defined as the minimum self‑information of a string with respect to a k‑th‑order Markov source, normalized by the string length. The paper explains that H₀(s) depends only on character frequencies, whereas Hₖ(s) for k ≥ 1 incorporates context of length k, yielding a monotone decreasing sequence bounded below by zero. It is shown that Hₖ₊₁(s) ≤ Hₖ(s) ≤ log σ, and that Hₖ(s)·|s| serves as a lower bound on the number of bits any algorithm using contexts of length ≤ k must use, by the noiseless coding theorem.

The authors illustrate the concept with the string “TORONTO”, calculating H₀ ≈ 1.84 bits/character, H₁ ≈ 0.29 bits/character, and higher orders equal to zero, demonstrating how context dramatically reduces uncertainty. They compare empirical entropy with stochastic complexity (minimum description length), noting that stochastic complexity allows non‑Markov sources and accounts for model cost but is more intricate to compute.

Two major applications are highlighted. First, Manzini’s 2001 analysis of the Burrows–Wheeler Transform (BWT) proved that BWT‑based compression stores a string in at most 8 Hₖ(s)·n + (μ + 2/25)n + σᵏ(2σ log σ + 9) bits for any k ≥ 0, providing a worst‑case bound that mirrors the probabilistic entropy but applies to any individual string. Second, Ferragina and Manzini’s 2005 work on compressed full‑text indexes showed that a string can be stored in 5 Hₖ(s)·n + o(n) bits while supporting pattern search in O(ℓ + o_cc log^{1+ε} n) time for any k ≥ 0 and ε > 0. These results sparked extensive research on entropy‑based data structures.

Finally, the paper acknowledges open problems and potential undiscovered applications of empirical entropy, emphasizing that despite its simplicity—requiring only a frequency table of k‑tuples—it offers a concrete, computable metric that bridges probability‑theoretic entropy and algorithmic compression performance without assuming a specific source model.