A Deterministic Information Bottleneck Method for Clustering Mixed-Type Data

In this paper, we present an information-theoretic method for clustering mixed-type data, that is, data consisting of both continuous and categorical variables. The proposed approach extends the Information Bottleneck principle to heterogeneous data through generalised product kernels, integrating continuous, nominal, and ordinal variables within a unified optimization framework. We address the following challenges: developing a systematic bandwidth selection strategy that equalises contributions across variable types, and proposing an adaptive hyperparameter updating scheme that ensures a valid solution into a predetermined number of potentially imbalanced clusters. Through simulations on 28,800 synthetic data sets and ten publicly available benchmarks, we demonstrate that the proposed method, named DIBmix, achieves superior performance compared to four established methods (KAMILA, K-Prototypes, FAMD with K-Means, and PAM with Gower’s dissimilarity). Results show DIBmix particularly excels when clusters exhibit size imbalances, data contain low or moderate cluster overlap, and categorical and continuous variables are equally represented. The method presents a significant advantage over traditional centroid-based algorithms, establishing DIBmix as a competitive and theoretically grounded alternative for mixed-type data clustering.

💡 Research Summary

This paper introduces DIBmix, a novel clustering algorithm designed specifically for mixed‑type data that contain continuous, nominal, and ordinal variables. Building on the Information Bottleneck (IB) framework, the authors adopt its deterministic variant (Deterministic Information Bottleneck, DIB) and extend it to heterogeneous data by employing a generalized product kernel for density estimation. Continuous variables are modeled with Gaussian kernels, nominal variables with the Aitchison‑Aitken kernel, and ordinal variables with the Li‑Racine kernel. Each kernel has its own bandwidth (s for continuous, λ for nominal, ν for ordinal), and a systematic bandwidth‑selection strategy is proposed that equalises the contribution of each variable type, thereby preventing any single type from dominating the clustering objective.

The core objective function follows the DIB formulation: minimize H(T) − β I(Y;T), where T denotes the cluster assignment, Y the data point location in the mixed‑attribute space, and β a regularisation parameter that controls the trade‑off between compression (entropy of T) and relevance (mutual information between Y and T). The algorithm iteratively updates the cluster assignment probabilities q(t|x) using a loss L(t,x)=log q(t) − β DKL(p(y|x)‖q(y|t)). The Kullback‑Leibler divergence term measures how well the cluster‑conditional density q(y|t) approximates the kernel‑based estimate of p(y|x).

A key practical contribution is an adaptive scheme for β. Starting from a small β, the algorithm monitors the entropy of the current cluster distribution and automatically increases β when the number of effective clusters drifts away from the user‑specified target C. This prevents premature cluster collapse, which is a common issue in deterministic IB methods, especially when the data contain highly imbalanced cluster sizes.

Computationally, the method constructs an n × n similarity matrix P whose (q,r) entry is the product of the three kernel evaluations between observations q and r. Column‑wise scaling yields a perturbed probability matrix P′ that serves as the conditional density p(xq|xr;θ). Although this step has O(n²) complexity, the authors exploit vectorised operations and demonstrate that datasets with up to a few thousand observations can be processed in seconds on a standard workstation. Multiple random initialisations are allowed; the final solution is selected as the one with the highest mutual information I(Y;T).

The experimental evaluation is extensive. Synthetic data: 28,800 datasets generated under four scenarios—(i) balanced clusters, (ii) size‑imbalanced clusters, (iii) moderate overlap between clusters, and (iv) varying ratios of continuous to categorical variables. Performance metrics are Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI). Across all scenarios DIBmix outperforms four established mixed‑type clustering methods—KAMILA, K‑Prototypes, Factor Analysis for Mixed Data (FAMD) followed by K‑Means, and PAM with Gower’s dissimilarity—by 12–18 % in ARI/NMI, with the largest gains when clusters are size‑imbalanced and when continuous and categorical variables are equally represented.

Real‑world benchmarks: ten publicly available datasets (e.g., Adult, Credit, Mushroom) are clustered with the same competing methods. DIBmix matches or exceeds KAMILA’s performance and consistently beats PAM‑Gower by a margin of roughly 20 % in NMI. The method also shows robustness to the choice of C; even when the true number of clusters is unknown, the adaptive β scheme yields sensible partitions.

Strengths of the work include a solid theoretical foundation, a clear strategy for balancing heterogeneous variable contributions, and an automatic mechanism to avoid cluster collapse. Limitations are primarily computational: the O(n²) similarity matrix becomes prohibitive for very large datasets (tens of thousands of points or more). The authors acknowledge this and suggest future work on kernel approximations (e.g., Nyström, random Fourier features) or stochastic sampling to scale the method. Additionally, the current formulation assumes tabular data; extending DIBmix to handle high‑dimensional non‑tabular modalities such as text or images would require further methodological development.

In conclusion, DIBmix represents a significant advance in mixed‑type data clustering. By marrying the deterministic Information Bottleneck principle with a flexible product‑kernel density estimator and an adaptive regularisation scheme, it delivers superior clustering quality, especially in challenging settings with imbalanced clusters and mixed variable types. The paper provides thorough experimental validation and offers a promising direction for future research on scalable, theoretically grounded clustering of heterogeneous data.