LIT-LVM: Structured Regularization for Interaction Terms in Linear Predictors using Latent Variable Models

Some of the simplest, yet most frequently used predictors in statistics and machine learning use weighted linear combinations of features. Such linear predictors can model non-linear relationships between features by adding interaction terms corresponding to the products of all pairs of features. We consider the problem of accurately estimating coefficients for interaction terms in linear predictors. We hypothesize that the coefficients for different interaction terms have an approximate low-dimensional structure and represent each feature by a latent vector in a low-dimensional space. This low-dimensional representation can be viewed as a structured regularization approach that further mitigates overfitting in high-dimensional settings beyond standard regularizers such as the lasso and elastic net. We demonstrate that our approach, called LIT-LVM, achieves superior prediction accuracy compared to the elastic net, hierarchical lasso, and factorization machines on a wide variety of simulated and real data, particularly when the number of interaction terms is high compared to the number of samples. LIT-LVM also provides low-dimensional latent representations for features that are useful for visualizing and analyzing their relationships.

💡 Research Summary

The paper addresses the long‑standing challenge of estimating interaction coefficients in linear models when all pairwise products of p features are included. In high‑dimensional regimes (p² ≫ n) traditional sparsity‑inducing penalties such as the lasso or elastic net are insufficient, and hierarchical approaches (e.g., hierarchical lasso) impose a heredity constraint but do not exploit any latent structure of the interaction matrix Θ.
The authors propose LIT‑LVM (Linear Interaction Terms with a Latent Variable Model), a structured regularization framework that assumes an approximate low‑dimensional representation of Θ. Each original feature j is associated with a latent vector z_j ∈ ℝᵈ (d < p). Two concrete instantiations are presented:

1. Low‑rank model – θ_{jk} ≈ z_jᵀz_k + ε_{jk}. This relaxes the exact low‑rank assumption of classic factorization machines (FM) by allowing an error term ε_{jk}, thereby accommodating approximate rather than exact low‑rank structure.

2. Latent‑distance model – θ_{jk} ≈ α₀ − ‖z_j − z_k‖² + ε_{jk}. Here a negative quadratic distance term encourages features with strong positive interactions to be placed close together in latent space, while negative interactions push them apart, offering an intuitive geometric interpretation.

The overall objective combines three components:

• Prediction loss (L_pred) – standard loss for the underlying linear predictor (MSE for regression, log‑loss for classification, negative partial log‑likelihood for Cox proportional hazards).
• Traditional regularization (L_reg) – elastic‑net‑type ℓ₁/ℓ₂ penalties on the main‑effect coefficients β and the flattened interaction vector θ_flat.
• Latent‑variable regularization (L_lvm) – a penalty that forces the estimated interaction matrix to be consistent with the chosen latent representation (e.g., Σ_{jk}(θ_{jk} − z_jᵀz_k)² or Σ_{jk}(θ_{jk} − α₀ + ‖z_j − z_k‖²)²).

Hyper‑parameters λ_r and λ_l control the strength of L_reg and L_lvm respectively. Optimization proceeds by alternating updates: first, β and θ_flat are updated using standard elastic‑net solvers; second, the latent vectors Z (and α₀ for the distance model) are refined by gradient‑based methods (e.g., Adam) to reduce L_lvm while keeping the current θ_fixed. This block‑coordinate descent iterates until convergence.

Empirical evaluation comprises extensive simulations and eight real‑world datasets spanning genomics, electronic health records, and advertising click‑through prediction. In simulations varying p/n ratios, noise levels, and the degree of underlying low‑dimensional structure, LIT‑LVM consistently outperforms (i) elastic‑net with all interactions, (ii) hierarchical lasso, and (iii) sparse factorization machines, achieving 3–7 % lower RMSE or higher AUC, especially when p² ≫ n. Real‑data experiments confirm these gains across both regression and classification tasks.

A notable by‑product is the learned latent embeddings Z. Visualizing Z with PCA or t‑SNE reveals coherent clusters that correspond to domain‑specific relationships. In a kidney‑transplant compatibility study, the latent space groups HLA alleles with similar compatibility profiles, uncovering clinically meaningful patterns not captured by standard coefficients.

Limitations include sensitivity to the choice of latent dimension d, higher computational cost relative to pure FM approaches (the alternating scheme requires repeated solves of elastic‑net subproblems), and potential bias if the true interaction matrix deviates strongly from any low‑dimensional approximation.

Future directions suggested are: (a) automated selection of d via Bayesian model evidence or cross‑validation, (b) scalable implementations using sparse matrix algebra and GPU acceleration, (c) extensions to higher‑order interactions (third‑order and beyond), and (d) integration of domain knowledge (e.g., known biological pathways) into the latent‑space prior.

In summary, LIT‑LVM introduces a principled way to impose an approximate low‑dimensional structure on interaction coefficients, bridging the gap between fully unstructured sparsity and exact low‑rank factorization. By jointly learning main effects, interaction weights, and latent feature embeddings, it delivers superior predictive performance and interpretable latent representations, marking a substantive advance over existing regularization techniques for interaction‑rich linear models.