Frequency Sensitive Duplicate Detection Using Multi-Metric Spaces

Classical metric spaces often fail to model data-intensive systems where repetition and frequency of values are meaningful. In applications such as transactional databases, sensor logs, and record linkage, conventional distance measures ignore multiplicity information, leading to information loss and incorrect similarity judgments. This paper introduces multi-metric spaces defined on multisets and valued in the multi-real number system, providing a principled way to incorporate frequency into distance computations. We demonstrate the usefulness of multi-metrics through a frequency sensitive duplicate detection example, showing improved accuracy over classical metric based approaches.

💡 Research Summary

The paper addresses a fundamental shortcoming of classical metric spaces when applied to data‑intensive domains where the frequency of attribute occurrences carries semantic weight. Traditional similarity measures—such as Hamming, Jaccard, cosine, or Euclidean distance—operate on sets or fixed‑length vectors and therefore collapse repeated elements into a single presence indicator or normalize them away. This loss of multiplicity leads to misleading similarity judgments, especially in duplicate detection, record linkage, and data cleaning tasks where two records may share the same distinct attributes but differ markedly in how often those attributes appear.

To remedy this, the authors introduce a mathematically rigorous framework built on multisets (also called m‑sets) and a novel number system called multi‑real numbers. A multiset M over a universe X is represented by a count function C_M : X → ℕ, where C_M(x) records how many times element x occurs. Standard multiset operations (union, intersection, addition ⊕, subtraction ⊖) are defined by pointwise max, min, sum, and non‑negative difference of the count functions. The support set M* (the set of elements with positive count) and cardinality |M| are also defined, and both bounded (