Applications of Universal Source Coding to Statistical Analysis of Time Series

We show how universal codes can be used for solving some of the most important statistical problems for time series. By definition, a universal code (or a universal lossless data compressor) can compress any sequence generated by a stationary and ergodic source asymptotically to the Shannon entropy, which, in turn, is the best achievable ratio for lossless data compressors. We consider finite-alphabet and real-valued time series and the following problems: estimation of the limiting probabilities for finite-alphabet time series and estimation of the density for real-valued time series, the on-line prediction, regression, classification (or problems with side information) for both types of the time series and the following problems of hypothesis testing: goodness-of-fit testing, or identity testing, and testing of serial independence. It is important to note that all problems are considered in the framework of classical mathematical statistics and, on the other hand, everyday methods of data compression (or archivers) can be used as a tool for the estimation and testing. It turns out, that quite often the suggested methods and tests are more powerful than known ones when they are applied in practice.

💡 Research Summary

The paper “Applications of Universal Source Coding to Statistical Analysis of Time Series” demonstrates how universal lossless data compressors—algorithms that asymptotically achieve the Shannon entropy for any stationary and ergodic source—can be directly employed as statistical tools for a wide range of problems in time‑series analysis. The authors begin by recalling the classic Laplace estimator for i.i.d. sequences and the later Krichevsky predictor, showing that both yield average Kullback–Leibler (KL) divergences that decay as O(log t / t) and O(log t / 2t) respectively. The Krichevsky predictor is proved to be minimax‑optimal among all predictors for i.i.d. sources, attaining the lower bound on the KL error.

Extending beyond i.i.d. data, the paper treats finite‑order Markov processes by viewing a Markov source of order m as a collection of |A|^m independent i.i.d. sub‑sources, thereby allowing the same universal predictors to be applied. However, the authors note a fundamental impossibility: no single predictor can guarantee vanishing pointwise KL error for the entire class of stationary ergodic processes. Instead, they introduce a predictor R whose Cesàro‑averaged KL error (the time‑average of the per‑step divergences) converges to zero with probability one for any stationary ergodic source. This average‑error perspective restores universality for a much broader class of processes.

For real‑valued time series, the authors propose quantization (partitioning the real line into a finite alphabet) and then applying the same universal coding machinery. The key insight is that the code length –log γ(x₁…x_t) produced by a compressor γ approximates the negative log‑likelihood –log P(x₁…x_t). Consequently, the redundancy (the excess code length over the true entropy) is exactly the KL divergence between the true distribution and the model implied by the compressor. Thus, any off‑the‑shelf compressor can be used for online density estimation: the shorter the compressed file, the closer the implied model is to the true distribution.

The paper then leverages this connection for hypothesis testing. Three testing problems are addressed: (1) goodness‑of‑fit (identity) testing, where the statistic is the difference between the compressed length under the hypothesized model and the length achieved by the universal compressor; (2) two‑sample (identity) testing, where the combined compression of two series is compared to the sum of their separate compressions; and (3) serial independence testing, where blockwise compression gains are examined to detect temporal dependence. In each case, the test statistic is a simple function of compressed file sizes, and the authors show empirically that these compression‑based tests outperform classical counterparts, such as the NIST suite for randomness testing, because modern compressors automatically capture a wide variety of latent regularities.

A practical implementation guide is provided: after appropriate preprocessing (alphabet mapping for categorical data, quantization for continuous data), one runs a standard archiver (e.g., zip, rar, arj) on the data, records the file sizes before and after compression, and computes the relevant differences. No specialized statistical software is required; the compressor itself serves as a universal estimator, predictor, and test statistic generator.

In conclusion, the work establishes a bridge between information theory and classical mathematical statistics: universal source coding is not merely a data‑compression technique but a universal statistical engine. By interpreting code lengths as log‑likelihoods, the authors obtain estimators, predictors, and hypothesis tests that are asymptotically optimal for broad classes of processes and often more powerful in finite‑sample practice. Future directions suggested include extensions to multivariate series, non‑stationary environments, and integration with modern deep‑learning based compressors, promising an even richer toolbox for statistical inference grounded in compression theory.