A New Combination of Preconditioned Gradient Descent Methods and Vector Extrapolation Techniques for Nonlinear Least-Squares Problems

Vector extrapolation methods are widely used in large-scale simulation studies, and numerous extrapolation-based acceleration techniques have been developed to enhance the convergence of linear and nonlinear fixed-point iterative methods. While classical extrapolation strategies often reduce the number of iterations or the computational cost, they do not necessarily lead to a significant improvement in the accuracy of the computed approximations. In this paper, we study the combination of preconditioned gradient-based methods with extrapolation strategies and propose an extrapolation-accelerated framework that simultaneously improves convergence and approximation accuracy. The focus is on the solution of nonlinear least-squares problems through the integration of vector extrapolation techniques with preconditioned gradient descent methods. A comprehensive set of numerical experiments is carried out to study the behavior of polynomial-type extrapolation methods and the vector $\varepsilon$-algorithm when coupled with gradient descent schemes, with and without preconditioning. The results demonstrate the impact of extrapolation techniques on both convergence rate and solution accuracy, and report iteration counts, computational times, and relative reconstruction errors. The performance of the proposed hybrid approaches is further assessed through a benchmarking study against Gauss–Newton methods based on generalized Krylov subspaces.

💡 Research Summary

The paper proposes a hybrid acceleration framework for solving large‑scale nonlinear least‑squares problems of the form ( \min_{x\in\mathbb{R}^n}|y-f(x)|_2^2 ). The core idea is to combine preconditioned gradient‑descent schemes with vector extrapolation techniques. Two preconditioned gradient methods are considered: (i) PGD, which uses the diagonal of the Jacobian (J_f(x_k)) as a preconditioner, and (ii) SGD, which uses the diagonal of the Gauss‑Newton approximation (J_f^T J_f). Both methods employ an Armijo‑type line search to guarantee sufficient decrease of the objective.

The extrapolation side employs three well‑known vector extrapolation algorithms: Reduced Rank Extrapolation (RRE), Minimal Polynomial Extrapolation (MPE), and the vector ε‑algorithm (VEA). Given a sequence of iterates (s_k) (e.g., the current solution estimates or gradients), the extrapolation constructs a new estimate (t_{k,q}) by forming linear combinations of the last (q+1) vectors. Mathematically, the extrapolated point can be written as
\