Information Distance in Multiples

Information distance is a parameter-free similarity measure based on compression, used in pattern recognition, data mining, phylogeny, clustering, and classification. The notion of information distance is extended from pairs to multiples (finite lists). We study maximal overlap, metricity, universality, minimal overlap, additivity, and normalized information distance in multiples. We use the theoretical notion of Kolmogorov complexity which for practical purposes is approximated by the length of the compressed version of the file involved, using a real-world compression program. {\em Index Terms}– Information distance, multiples, pattern recognition, data mining, similarity, Kolmogorov complexity

💡 Research Summary

The paper extends the well‑known compression‑based information distance from pairs of objects to finite lists (multiples). Starting from Kolmogorov complexity K(x) and its conditional version K(x|y), the authors define for a list X = (x₁,…,x_m) two central quantities:

• E_max(X) = max_{x∈X} K(X|x) – the length of the shortest program that, given any element of the list, can reconstruct the whole list. This is interpreted as the information contained in the “most comprehensive” object.
• E_min(X) = min_{x∈X} K(X|x) – the length of the shortest program that, given the best element, can reconstruct the list. This corresponds to the “most specialized” object.

The paper’s main contributions are a series of theoretical results that mirror those known for the pairwise case but hold for arbitrary list sizes, often with simpler proofs.

1. Maximal Overlap (Theorem 3.1). The authors show that a single program of length k₁ = E_min(X) plus a short “overlap” string of length ℓ = E_max – E_min (plus logarithmic overhead) suffices to reconstruct the list from any element. In other words, the information needed to go from any x_i to any x_k can be split into a common core (k₁ bits) and a per‑element remainder (ℓ bits). This theorem directly yields the earlier result (I.1) from the cited work and clarifies the interpretation of E_min as a single program representing maximal overlap.

2. Metricity (Theorem 4.1). E_max satisfies the three metric axioms—positivity, symmetry, and the triangle inequality—up to an additive O(log K) term, where K is the largest Kolmogorov complexity involved. The proof adapts the pairwise argument by using the maximal‑overlap decomposition and standard properties of Kolmogorov complexity.

3. Universality (Theorem 5.2). The authors introduce the notion of an “admissible list distance” (total, possibly asymmetric, upper‑semicomputable, and satisfying a density condition). They prove that E_max is the minimal such distance: any admissible distance D must dominate E_max up to a constant. Hence E_max is the most “universal” similarity measure for lists, just as the pairwise information distance is universal for two objects.

4. Additivity (Theorem 6.1) and Minimal Overlap (Theorem 7.1). These results extend the additive property of information distance (the distance between concatenated objects is bounded by the sum of individual distances) and the existence of a program achieving the minimal overlap, respectively, to the list setting.

5. Non‑Metric Normalized Information Distance for Lists. While the normalized information distance (NID) is a metric for pairs, the authors demonstrate that any straightforward extension to lists of size ≥ 3 fails the triangle inequality. They discuss why the usual normalizing factor (max{K(x_i)}) is insufficient and suggest that new normalizations are required for multi‑object similarity.

6. Practical Approximation. Since Kolmogorov complexity is incomputable, the paper follows the standard practice of approximating K(·) by the length of a compressed file using real‑world compressors (gzip, bzip2, etc.). The authors argue that this approximation retains the theoretical properties sufficiently for practical pattern‑recognition, clustering, phylogeny, and classification tasks.

7. Applications and Outlook. The extended framework is motivated by real‑world scenarios where one wishes to extract a “most comprehensive” summary (E_min) or a “most specialized” representative (E_max) from a collection of documents, reviews, sensor logs, or biological sequences. The paper suggests that the new theory can improve heterogeneous data clustering, anomaly detection, and semantic analysis. Future work is outlined: designing proper normalizations for lists, studying the impact of different compressors, and scaling the algorithms to massive datasets.

In summary, the paper provides a solid theoretical foundation for information distance on multiples, proving metricity, universality, and additive properties, while also highlighting the challenges of normalizing the distance for more than two objects. By linking the abstract Kolmogorov framework to practical compression‑based approximations, it opens the door to a wide range of multi‑object similarity applications.