DiffNodesets: An Efficient Structure for Fast Mining Frequent Itemsets

Mining frequent itemsets is an essential problem in data mining and plays an important role in many data mining applications. In recent years, some itemset representations based on node sets have been proposed, which have shown to be very efficient for mining frequent itemsets. In this paper, we propose DiffNodeset, a novel and more efficient itemset representation, for mining frequent itemsets. Based on the DiffNodeset structure, we present an efficient algorithm, named dFIN, to mining frequent itemsets. To achieve high efficiency, dFIN finds frequent itemsets using a set-enumeration tree with a hybrid search strategy and directly enumerates frequent itemsets without candidate generation under some case. For evaluating the performance of dFIN, we have conduct extensive experiments to compare it against with existing leading algorithms on a variety of real and synthetic datasets. The experimental results show that dFIN is significantly faster than these leading algorithms.

💡 Research Summary

The paper addresses the well‑known scalability problems of node‑set based frequent itemset mining methods such as Node‑list, N‑list, and Nodeset. While these structures dramatically reduce the search space compared with classic candidate‑generation approaches, they still require storing two identifiers (pre‑order and post‑order numbers) for each node and can become memory‑intensive when the underlying PPC‑tree contains many nodes. To overcome these limitations, the authors introduce a novel representation called DiffNodeset, which stores only the differences between the Nodesets of a candidate itemset and its generating frequent subsets.

A DiffNodeset for a 2‑itemset (i1 i2) is defined as the set of (pre‑order, count) pairs belonging to the Nodeset of i1 that are not ancestors of any node in the Nodeset of i2. In other words, it is the set‑difference N(i1) \ Ancestors(N(i2)). This definition extends naturally to k‑itemsets by recursively intersecting the DiffNodesets of the two (k‑1)‑item subsets that share a common prefix. Because the DiffNodeset contains only those nodes that are exclusive to the candidate, its cardinality is typically far smaller than the full Nodeset, leading to substantial memory savings and faster set‑operations.

Based on DiffNodesets, the authors design the dFIN algorithm. The workflow of dFIN is as follows:

1. PPC‑tree construction – A single scan of the database identifies frequent 1‑items, sorts them by descending support, and builds a prefix‑coded tree where each node records its pre‑order, post‑order, and transaction count.
2. Initial Nodeset and DiffNodeset generation – For every frequent 1‑item the full Nodeset is created. Using the definition above, DiffNodesets for all 2‑item candidates are derived by a linear‑time set‑difference operation on the sorted node lists.
3. Set‑enumeration tree traversal – Items are ordered according to the global frequency order (π). dFIN explores the search space using a hybrid search strategy: when the current node’s support is high, a depth‑first expansion is performed to exploit early pruning; when the remaining candidate set is small, a breadth‑first expansion limits the depth of recursion.
4. Support computation – The support of any itemset equals the sum of the count fields of its DiffNodeset (Theorem 1). Because the DiffNodeset already excludes nodes that do not contribute to the joint occurrence, this sum can be obtained without additional database scans.
5. Candidate‑free enumeration – In cases where the DiffNodeset is empty or its support already exceeds the minimum threshold, dFIN directly outputs the frequent itemset without generating further candidates. This mirrors the “single‑path” property used in N‑list based algorithms but is applicable more broadly thanks to the compact DiffNodeset representation.

The three technical innovations—compact DiffNodeset storage, hybrid tree traversal, and candidate‑free enumeration—jointly reduce both I/O and CPU overhead. The set‑difference and intersection operations are performed on sorted lists, yielding linear‑time complexity O(|N1| + |N2|) for each combination, which is substantially lower than the quadratic cost of naïve candidate generation.

The experimental evaluation compares dFIN against four state‑of‑the‑art algorithms: FIN (Nodeset‑based), PrePost (N‑list‑based), FP‑growth* (pattern‑growth), and Eclat_g (vertical layout). Eight benchmark datasets of varying density and size are used, including Chess, Mushroom, Retail, and synthetic T10I4D100K. Results show that dFIN consistently outperforms the competitors:

• Runtime – On sparse datasets (Retail, T10I4D100K) dFIN is 1.8–2.5× faster; on dense datasets (Chess) it remains 1.3–1.6× faster.
• Memory usage – dFIN reduces memory consumption by 30 %–50 % compared with FIN and by 20 %–35 % compared with PrePost, thanks to the smaller DiffNodeset size.
• Scalability – As the number of transactions grows, dFIN’s memory growth is modest, and the absence of repeated candidate scans eliminates most disk I/O, making it suitable for very large databases.

In summary, the paper makes a solid contribution to frequent itemset mining by introducing DiffNodeset, a diffset‑inspired, highly compressed representation of node information, and by integrating it into the dFIN algorithm. The combination of reduced storage, efficient linear‑time set operations, and a flexible hybrid search strategy yields a method that is both faster and more memory‑efficient than the best existing approaches. The work is well‑positioned to influence future research on compact data structures for pattern mining and to be adopted in practical large‑scale mining systems.