Nonlinear Sparse Bayesian Learning Methods with Application to Massive MIMO Channel Estimation with Hardware Impairments

Accurate channel estimation is critical for realizing the performance gains of massive multiple-input multiple-output (MIMO) systems. Traditional approaches to channel estimation typically assume ideal receiver hardware and linear signal models. However, practical receivers suffer from impairments such as nonlinearities in the low-noise amplifiers and quantization errors, which invalidate standard model assumptions and degrade the estimation accuracy. In this work, we propose a nonlinear channel estimation framework that models the distortion function arising from hardware impairments using Gaussian process (GP) regression while leveraging the inherent sparsity of massive MIMO channels. First, we form a GP-based surrogate of the distortion function, employing pseudo-inputs to reduce the computational complexity. Then, we integrate the GP-based surrogate of the distortion function into newly developed enhanced sparse Bayesian learning (SBL) methods, enabling distortion-aware sparse channel estimation. Specifically, we propose two nonlinear SBL methods based on distinct optimization objectives, each offering a different trade-off between estimation accuracy and computational complexity. Numerical results demonstrate significant gains over the Bussgang linear minimum mean squared error estimator and linear SBL, particularly under strong distortion and at high signal-to-noise ratio.

💡 Research Summary

This paper tackles the problem of uplink channel estimation in massive multiple‑input multiple‑output (MIMO) systems when the receiver hardware is non‑ideal. Conventional estimators such as linear minimum mean‑square error (LMMSE) or sparse Bayesian learning (SBL) assume ideal front‑ends and either ignore the distortion or treat it with a Bussgang linearisation, which only captures first‑ and second‑order statistics of the nonlinearity. The authors propose a fundamentally different approach: they learn the unknown distortion function directly from data using Gaussian‑process (GP) regression, and then embed this learned surrogate into an enhanced SBL framework that exploits the angular sparsity of massive MIMO channels.

Key technical ingredients

1. GP‑based distortion modelling – The distortion function (g(\cdot)) that maps the ideal received signal (z) to the distorted observation (y) is treated as a smooth, unknown mapping. Paired samples ({z_i, y_i}) obtained during pilot transmission are used to train a GP. To keep the computational burden tractable, the authors employ pseudo‑inputs (also called inducing points), reducing the cubic complexity of standard GP inference to a manageable level (roughly (\mathcal{O}(M\tilde M^2)) where (\tilde M) is the number of pseudo‑inputs, typically 20‑30).

2. Enhanced SBL with GP surrogate – In the linear SBL formulation the channel vector (u) is estimated together with a set of precision (hyper‑)parameters (\lambda) under a Gaussian‑Gamma hierarchical prior. When the observation model becomes (y = g(Au) + e) with a nonlinear (g), the authors propose two variants:

   • Marginal‑Posterior SBL (MPSBL) – Maximises the marginal posterior of ((u,\lambda)) after integrating out the GP latent function. The GP mean and Jacobian are used to linearise the observation around the current iterate, leading to a sequence of approximate linear systems that are solved efficiently.
   • Joint‑Posterior SBL (JPSBL) – Directly maximises the joint posterior over the channel, the GP function, and the hyper‑parameters. This requires a Newton‑type optimisation of a nonlinear system at each iteration, offering higher accuracy at the cost of extra computation.

3. Complexity analysis and practical guidelines – The authors derive explicit expressions for the per‑iteration cost of both algorithms and study how the number and placement of pseudo‑inputs affect NMSE. They show that a modest number of pseudo‑inputs yields a good trade‑off between accuracy and runtime, and they provide recommendations for initializing the algorithms (e.g., using the linear BLMMSE estimate as a warm start).

4. Extensions – The framework is readily adapted to hybrid analog‑digital beamforming (by modifying the forward model) and to extreme quantisation such as 1‑bit ADCs. For the latter, the likelihood is changed from Gaussian to a Bernoulli model, and the GP inference is performed with a probit‑type approximation.

5. Numerical evaluation – Simulations cover a wide range of system parameters: SNR from 0 to 30 dB, pilot lengths, numbers of BS antennas (64–256), numbers of users (8–16), and channel sparsity levels (3–8 dominant paths). The distortion is modelled as a third‑order LNA nonlinearity (g(z)=z - a|z|^2z) with controllable strength (\alpha). Results show that both MPSBL and JPSBL substantially outperform least‑squares, Bussgang‑based LMMSE (BLMMSE), and conventional linear SBL. Gains of 3–6 dB in NMSE are observed especially when (\alpha) is large (strong nonlinearity) and the SNR is high (>20 dB). The performance gap widens with shorter pilot sequences, confirming the robustness of the proposed methods under severe training overhead.

Conclusions

The paper delivers a novel, data‑driven solution for channel estimation in massive MIMO systems plagued by hardware impairments. By learning the distortion function with a scalable GP surrogate and integrating it into a sparsity‑exploiting Bayesian estimator, the authors bridge the gap between idealised theory and practical hardware‑constrained deployments. The two proposed SBL variants provide flexible options: a lower‑complexity marginal‑posterior method suitable for real‑time processing, and a higher‑accuracy joint‑posterior method for offline or high‑performance scenarios. The comprehensive analysis, clear complexity‑accuracy trade‑offs, and extensions to hybrid beamforming and 1‑bit ADCs make this work a valuable reference for researchers and engineers developing next‑generation (6G/7G) massive MIMO receivers where non‑ideal hardware cannot be ignored.