A Family of LZ78-based Universal Sequential Probability Assignments

We propose and study a family of universal sequential probability assignments on individual sequences, based on the incremental parsing procedure of the Lempel-Ziv (LZ78) compression algorithm. We show that the normalized log loss under any of these models converges to the normalized LZ78 codelength, uniformly over all individual sequences. To establish the universality of these models, we consolidate a set of results from the literature relating finite-state compressibility to optimal log-loss under Markovian and finite-state models. We also consider some theoretical and computational properties of these models when viewed as probabilistic sources. Finally, we present experimental results showcasing the potential benefit of using this family – as models and as sources – for compression, generation, and classification.

💡 Research Summary

This paper proposes and thoroughly investigates a novel family of universal sequential probability assignments (SPAs) for individual sequences, rooted in the incremental parsing mechanism of the Lempel-Ziv 1978 (LZ78) compression algorithm.

The core idea is to leverage the context naturally defined by the LZ78 parsing process. As LZ78 parses a sequence into phrases, it builds a dynamic prefix tree where each node represents a distinct context—the path from the root to that node. The proposed SPA family operates by conditioning the probability of the next symbol on its current LZ78 context (the node being traversed when that symbol is parsed). For each context node z, the probability distribution for the next symbol is formulated as a Bayesian mixture: it combines the empirical distribution of symbols historically observed after context z with an arbitrary Dirichlet prior distribution over the alphabet. By varying the parameters of this Dirichlet prior, a broad family of models is defined, generalizing several earlier LZ78-based prediction schemes.

The paper establishes two key theoretical results. First, it proves that for any model within this family, the normalized cumulative log-loss (self-entropy) incurred on any individual sequence converges uniformly to the normalized LZ78 codeword length of that sequence. This directly ties the predictive performance of these probability models to the fundamental compression efficiency of LZ78. Second, the paper rigorously demonstrates the universality of this model family. It does so by consolidating known results from the literature to show the asymptotic equivalence between finite-state compressibility, optimal log-loss achievable by Markov models, and optimal log-loss achievable by finite-state machines. Building on this equivalence, the authors prove that the LZ78-based SPA family is universal in the sense that its asymptotic performance is never worse than that of any finite-state machine-based SPA.

Beyond its role as a predictive model, the paper explores interpreting the LZ78 SPA as a probabilistic source for generating new sequences. Sampling from this source involves traversing the LZ78 tree and generating symbols according to the posterior predictive distribution at each node. This perspective opens up applications in generation and provides a link to recent work on universal lossy compression using random codebooks weighted by LZ78 complexity.

The authors also discuss computational aspects, noting that maintaining the LZ78 tree and updating counts can be done in linear time with respect to the sequence length, making the approach practically feasible. Preliminary experimental results are presented to showcase the potential utility of the proposed family, both as a model for classification tasks (by using sequence likelihood as a feature) and as a source for data generation. The work bridges foundational concepts from universal compression, sequential prediction, and Bayesian estimation, offering a framework with strong theoretical guarantees and practical efficiency.