Frequency Moments in Noisy Streaming and Distributed Data under Mismatch Ambiguity

We propose a novel framework for statistical estimation on noisy datasets. Within this framework, we focus on the frequency moments ($F_p$) problem and demonstrate that it is possible to approximate $F_p$ of the unknown ground-truth dataset using sublinear space in the data stream model and sublinear communication in the coordinator model, provided that the approximation ratio is parameterized by a data-dependent quantity, which we call the $F_p$-mismatch-ambiguity. We also establish a set of lower bounds, which are tight in terms of the input size. Our results yield several interesting insights: (1) In the data stream model, the $F_p$ problem is inherently more difficult in the noisy setting than in the noiseless one. In particular, while $F_2$ can be approximated in logarithmic space in terms of the input size in the noiseless setting, any algorithm for $F_2$ in the noisy setting requires polynomial space. (2) In the coordinator model, in sharp contrast to the noiseless case, achieving polylogarithmic communication in the input size is generally impossible for $F_p$ under noise. However, when the $F_p$ mismatch ambiguity falls below a certain threshold, it becomes possible to achieve communication that is entirely independent of the input size.

💡 Research Summary

The paper introduces a novel framework for estimating statistical functions on noisy datasets, focusing on the p‑th frequency moment (Fₚ) problem. In this setting each observed item σᵢ is a noisy version of an unknown ground‑truth item τᵢ drawn from a hidden universe U. The only access to the underlying similarity structure is an oracle that, given two observed items, answers whether they are “similar”. This oracle may produce both false positives and false negatives, which are captured by constructing a similarity graph Gσ whose vertices correspond to stream positions and edges indicate similarity according to the oracle. For each vertex i, Bσᵢ denotes the set of indices similar to σᵢ; the corresponding true cluster is Bτᵢ. The authors define a data‑dependent parameter called the Fₚ‑mismatch‑ambiguity ηₚ, which measures the aggregate discrepancy between Bσᵢ and Bτᵢ across all i, scaled by the true moment Fₚ(τ). When the data are noiseless ηₚ = 0; for p = 1 the parameter is always zero, while for p ≥ 2 it reflects the number of false‑positive and false‑negative edges in a non‑linear fashion.

The work studies two canonical big‑data models:

1. Data‑stream model – Items arrive sequentially; the algorithm may make one or a constant number of passes and must use sublinear memory.
   Algorithm: Assuming ηₚ ≤ 1/(3·p!), the authors give a one‑pass (ε + O(ηₚ))-approximation algorithm that uses O((1/ε²)·m^{1‑1/p}) words of space, where m is the stream length. The algorithm essentially adapts classic AMS‑type sketches but incorporates ηₚ into the error analysis.
   Lower bound: For any constant‑pass algorithm achieving (ε + C·ηₚ)-approximation (C ≥ 0), at least Ω((1/ε^{1/p})·m^{1‑1/p}) bits of memory are required. Consequently, for p = 2 the classic logarithmic‑space solution for noiseless streams becomes impossible under noise; polynomial space is necessary.

2. Coordinator (distributed) model – The dataset is partitioned across k sites, each communicating with a central coordinator in rounds.
   Algorithm (2 rounds): When ηₚ ≤ 0.4, a two‑round protocol yields a (ε + O(ηₚ))-approximation using O((1/ε²)·k·m^{1‑1/p}) words of communication.
   Lower bound: For any constant‑C ≥ 0, any (ε + C·ηₚ)-approximation algorithm must communicate at least Ω((1/ε^{1/p})·m^{1‑1/p}) bits when the number of sites satisfies k ≥ 10·p·(C+1). This shows that, unlike the noiseless case where polylogarithmic communication suffices, noise forces a dependence on the input size.
   Small‑ambiguity regime: If ηₚ is sufficiently tiny—specifically ηₚ ≤ ε·p⁴·k^{-(p‑1)}—the authors present a three‑round protocol whose communication cost drops to O(k²/ε³) for p = 2, independent of m. This reveals a phase transition: once the mismatch ambiguity falls below a threshold, the communication cost matches that of the noiseless setting.

The paper also observes that ηₚ coincides (up to scaling) with the optimal correlation‑clustering cost of the similarity graph, linking the statistical estimation problem to a well‑studied clustering objective. Consequently, existing clustering algorithms could be employed to estimate or reduce ηₚ before applying the moment‑estimation protocols.

Overall contributions:

• Introduction of the Fₚ‑mismatch‑ambiguity ηₚ as a quantitative measure of noise impact on frequency‑moment estimation.
• Sublinear‑space streaming algorithms whose error gracefully degrades with ηₚ, together with matching space lower bounds.
• Sublinear‑communication distributed algorithms that achieve (ε + O(ηₚ))-approximation, plus tight communication lower bounds, and a special regime where communication becomes independent of the dataset size.
• Conceptual connection between statistical estimation under noise and correlation clustering, suggesting practical preprocessing pathways.

The results broaden the theory of streaming and distributed computation to realistic noisy environments such as near‑duplicate detection in search logs, duplicate image/video collections, or ambiguous outputs from large language models. By quantifying how much “mismatch” can be tolerated before sublinear resources become insufficient, the work provides both a theoretical benchmark and a practical guideline for system designers handling massive, imperfect data streams.