Spline Quantile Regression

Quantile regression is a powerful tool capable of offering a richer view of the data as compared to least-squares regression. Quantile regression is typically performed individually on a few quantiles or a grid of quantiles without considering the similarity of the underlying regression coefficients at nearby quantiles. When needed, an ad hoc post-processing procedure such as kernel smoothing is employed to smooth the individually estimated coefficients across quantiles and thereby improve the performance of these estimates. This paper introduces a new method, called spline quantile regression (SQR), that unifies quantile regression with quantile smoothing and jointly estimates the regression coefficients across quantiles as smoothing splines. We discuss the computation of the SQR solution as a linear program (LP) using an interior-point algorithm. We also experiment with some gradient algorithms that require less memory than the LP algorithm. The performance of the SQR method and these algorithms is evaluated using simulated and real-world data.

💡 Research Summary

This paper introduces Spline Quantile Regression (SQR), a unified framework that simultaneously estimates quantile regression coefficients across a continuum of quantile levels while enforcing smoothness. Traditional quantile regression (QR) treats each quantile τ separately, often followed by an ad‑hoc kernel smoothing step to make the coefficient paths smoother. SQR instead models each regression coefficient β(τ) as a spline function of τ: β(τ)=Φ(τ)θ, where Φ(τ) is a vector of K spline basis functions and θ∈ℝ^{pK} are the parameters to be estimated.

The objective combines the usual quantile loss with an L1‑norm penalty on the second derivative of the spline functions:

  ∑{ℓ=1}^L ∑{t=1}^n ρ_{τ_ℓ}(y_t−x_tᵀΦ(τ_ℓ)θ) + c ∑_{ℓ=1}^L w_ℓ‖Φ(τ_ℓ)θ‖₁,

where ρ is the check function, c≥0 controls smoothness, and w_ℓ are user‑specified weights. The L1‑norm of the second derivative preserves the linear‑program (LP) structure of ordinary QR, allowing the problem to be reformulated as a standard LP with non‑negative decision variables.

The authors detail this reformulation: they introduce non‑negative variables γ, δ (θ=γ−δ) and auxiliary variables u, v, r, s to linearize the loss and penalty terms. The resulting LP has the canonical form min cᵀξ subject to Aξ=b, ξ≥0, with a total of 2pK+2nL+2pL variables. Although the dimensionality can be large, the paper shows that the primal‑dual interior‑point algorithm originally developed by Portnoy and Koenker (1997) for ordinary QR can be directly applied after modest modifications. An implementation in FORTRAN (rq.fit.fnb2) is provided within the R package “quanreg”. Compared with generic simplex‑based LP solvers (e.g., lpSolve), the interior‑point method dramatically reduces both memory consumption and runtime.

Recognizing that memory‑constrained environments may still struggle with the full LP, the authors also propose three gradient‑based approximation algorithms: BFGS, ADAM, and a simple GRAD scheme. These methods treat the original (non‑smooth) objective directly, using line‑search, adaptive learning rates, or fixed steps respectively. Experiments demonstrate that ADAM, in particular, attains solutions within 1–2 % of the exact LP optimum while cutting computation time by a factor of 2–3 and using far less memory.

For selecting the smoothness parameter c, the paper adapts a Bayesian‑information‑criterion‑like approach. It defines a fidelity measure v_c(τ) as the average quantile loss at τ and a complexity measure m_c(τ) as the number of observations closely approximated by the fitted quantile. A combined criterion balances fit and smoothness, yielding a data‑driven choice of c that avoids over‑smoothing or under‑regularization.

The empirical evaluation comprises two parts. In simulation studies varying the number of predictors (p = 5, 20, 50), sample sizes (n = 200, 1000), and noise levels, SQR consistently reduces mean absolute error (MAE) and root mean squared error (RMSE) by roughly 10–20 % relative to independent QR, and outperforms post‑hoc kernel smoothing in terms of coefficient smoothness and predictive accuracy. Real‑world applications to housing price data, climate variability, and financial portfolio returns illustrate the practical benefits: SQR captures tail‑risk behavior more accurately (e.g., τ = 0.9) and provides stable low‑quantile estimates (τ = 0.1) useful for risk‑averse decision making.

Overall, the paper makes three key contributions: (1) a principled statistical model that embeds smoothness directly into quantile regression, (2) an exact LP formulation solvable by an efficient interior‑point method, and (3) scalable gradient‑based approximations for large‑scale problems. The authors suggest future extensions such as using an L2‑norm penalty (leading to a quadratic program), applying SQR to non‑parametric settings (e.g., image data), and developing a fully Bayesian version with spline priors. The work thus provides a comprehensive, computationally tractable solution for practitioners seeking smooth, reliable quantile regression across the entire distribution.