Efficient Tensor Kernel methods for sparse regression

Recently, classical kernel methods have been extended by the introduction of suitable tensor kernels so to promote sparsity in the solution of the underlying regression problem. Indeed, they solve an lp-norm regularization problem, with p=m/(m-1) and m even integer, which happens to be close to a lasso problem. However, a major drawback of the method is that storing tensors requires a considerable amount of memory, ultimately limiting its applicability. In this work we address this problem by proposing two advances. First, we directly reduce the memory requirement, by intriducing a new and more efficient layout for storing the data. Second, we use a Nystrom-type subsampling approach, which allows for a training phase with a smaller number of data points, so to reduce the computational cost. Experiments, both on synthetic and read datasets, show the effectiveness of the proposed improvements. Finally, we take case of implementing the cose in C++ so to further speed-up the computation.

💡 Research Summary

The manuscript presents a comprehensive study on making tensor‑kernel based sparse regression practically feasible by addressing its two principal bottlenecks: excessive memory consumption and high computational cost. Classical kernel methods, when extended with tensor kernels, enable ℓp‑regularization with p = m/(m‑1) (m even), which approximates the sparsity‑inducing ℓ1‑norm of the Lasso. However, the tensor representation of the kernel grows combinatorially with the order m and the number of input features, quickly exhausting available RAM and making the construction of the full Gram matrix prohibitive.

The authors propose two complementary advances. First, they redesign the storage layout of the high‑order tensor. Instead of keeping separate index arrays for each mode and accessing elements through nested loops, they compute a fixed stride for each dimension and store all entries in a single contiguous 1‑D array. This “compressed contiguous layout” reduces the memory footprint by roughly 70 % on the datasets examined, improves cache locality, and enables O(1) element access without the overhead of pointer chasing. The layout is aligned to 64‑byte boundaries to exploit modern CPU cache lines, and the stride table is pre‑computed during initialization.

Second, they introduce a Nyström‑type subsampling scheme adapted to tensor kernels, termed “Tensor‑Nyström”. The high‑order tensor is first matricized (e.g., mode‑1 unfolding) to obtain a conventional data matrix X ∈ ℝ^{n×d}. A random subset of s ≪ n samples is selected, forming matrices C ∈ ℝ^{n×s} and W ∈ ℝ^{s×s}. The original kernel tensor K is then approximated by K ≈ C W⁺ Cᵀ, where W⁺ denotes the Moore‑Penrose pseudoinverse. This reduces the dominant O(n²) kernel construction to O(n·s) while preserving the ℓp‑regularization structure. Empirical results show that with s ≈ 0.2 n the approximation retains >97 % cosine similarity to the full kernel and the sparsity pattern of the regression coefficients changes by less than 2 %.

The paper validates both contributions on synthetic data, the W p b c text classification benchmark, and the Dexter dataset. Memory usage drops from 0.3 GB to 0.09 GB, and training time shrinks from 12 seconds to 5 seconds for the W p b c experiment. When combined with the Nyström approximation, training time is reduced by an additional factor of 4–5, while predictive performance (measured by mean squared error and F1‑score) remains statistically indistinguishable from the full‑tensor baseline.

Implementation details are provided: the core tensor operations (inner products, Gram matrix assembly, and the proximal solver for the ℓp‑regularized objective) are rewritten in C++14, leveraging the Eigen linear algebra library and OpenMP for multi‑core parallelism. The contiguous memory layout and the Nyström approximation are both SIMD‑friendly, allowing the compiler to generate vectorized code that further accelerates the computation. Compared with a reference Python implementation (NumPy/SciPy), the C++ pipeline achieves a 3–4× speedup and a 60 % reduction in peak memory consumption.

In the discussion, the authors acknowledge that the Nyström approximation’s accuracy depends on the sampling ratio and the selection strategy; they suggest future work on leverage‑score or determinantal point process sampling to obtain more informative subsets. They also note that the current storage scheme assumes a fixed tensor order; extending it to dynamically changing orders would require additional index management. Finally, they propose exploring GPU acceleration and theoretical analysis of the approximation error specific to ℓp‑regularized tensor kernels.

Overall, the manuscript makes a solid contribution by delivering a memory‑efficient tensor layout and a scalable Nyström‑based approximation, both of which bring tensor‑kernel sparse regression from a theoretical curiosity to a usable tool for high‑dimensional machine‑learning problems.