Fast Set Intersection in Memory

Set intersection is a fundamental operation in information retrieval and database systems. This paper introduces linear space data structures to represent sets such that their intersection can be computed in a worst-case efficient way. In general, given k (preprocessed) sets, with totally n elements, we will show how to compute their intersection in expected time O(n/sqrt(w)+kr), where r is the intersection size and w is the number of bits in a machine-word. In addition,we introduce a very simple version of this algorithm that has weaker asymptotic guarantees but performs even better in practice; both algorithms outperform the state of the art techniques in terms of execution time for both synthetic and real data sets and workloads.

💡 Research Summary

**
The paper addresses the problem of intersecting multiple static sets that reside entirely in main memory, a scenario common in modern search engines, data‑mining pipelines, and real‑time analytics. Traditional approaches—sorted posting lists (inverted indexes), adaptive comparison‑based algorithms, hierarchical structures (skip‑lists, trees), and hash‑based methods—either suffer from linear‑time scans when set sizes differ greatly, require many comparisons without translating into speed, or need complex bit‑manipulation that is costly in practice. Moreover, many of these techniques assume that the full intersection is large relative to the input sets, an assumption that does not hold for typical web‑search queries.

The authors propose a new framework that exploits two observations: (1) a machine word of w bits can represent a subset of a universe of size w as a bit‑mask, allowing the intersection of two such subsets to be computed with a single bitwise‑AND in O(1) time; (2) in real‑world workloads the size r of the final intersection is usually orders of magnitude smaller than the smallest input set. To combine these ideas, each input set is first sorted and then partitioned into blocks of size √w elements. Each block is hashed with a universal hash function h: Σ →