Are Deep Learning Based Hybrid PDE Solvers Reliable? Why Training Paradigms and Update Strategies Matter

Deep learning-based hybrid iterative methods (DL-HIMs) integrate classical numerical solvers with neural operators, utilizing their complementary spectral biases to accelerate convergence. Despite this promise, many DL-HIMs stagnate at false fixed points where neural updates vanish while the physical residual remains large, raising questions about reliability in scientific computing. In this paper, we provide evidence that performance is highly sensitive to training paradigms and update strategies, even when the neural architecture is fixed. Through a detailed study of a DeepONet-based hybrid iterative numerical transferable solver (HINTS) and an FFT-based Fourier neural solver (FNS), we show that significant physical residuals can persist when training objectives are not aligned with solver dynamics and problem physics. We further examine Anderson acceleration (AA) and demonstrate that its classical form is ill-suited for nonlinear neural operators. To overcome this, we introduce physics-aware Anderson acceleration (PA-AA), which minimizes the physical residual rather than the fixed-point update. Numerical experiments confirm that PA-AA restores reliable convergence in substantially fewer iterations. These findings provide a concrete answer to ongoing controversies surrounding AI-based PDE solvers: reliability hinges not only on architectures but on physically informed training and iteration design.

💡 Research Summary

This paper investigates the reliability of deep learning‑based hybrid iterative methods (DL‑HIMs) for solving large‑scale linear systems that arise from discretized partial differential equations (PDEs). DL‑HIMs combine a classical stationary smoother (e.g., damped Jacobi or Gauss‑Seidel) with a neural operator (such as DeepONet or a Fourier neural solver, FNS). The motivation is that classical smoothers efficiently damp high‑frequency error components, while neural operators exhibit a spectral bias toward low‑frequency modes; together they could accelerate convergence dramatically.

The authors identify a fundamental obstacle: “false fixed points.” In a DL‑HIM iteration the update δ = Gθ(u) − u may become (numerically) zero even though the physical residual r = f − Au remains large. This occurs because the spectral gap between the smoother and the neural operator can cause both components to produce negligible corrections for certain error modes. Consequently, the iteration appears to have converged (the mathematical fixed point of Gθ is reached) while the PDE solution has not been attained.

To understand and remedy this phenomenon, the paper provides a rigorous operator‑theoretic framework. It defines error‑propagation operators for the smoother (E_Sⁿ = (I − ωSA)ⁿ) and for the neural correction (E_Nθ(e) = e − Nθ(Ae)), as well as analogous residual‑propagation operators (R_Sⁿ, R_Nθ). When the neural operator is nonlinear, the combined error operator E = E_Nθ∘E_Sⁿ becomes state‑dependent, breaking the simple spectral‑radius analysis that underlies classical convergence theory.

A central contribution is the systematic study of training paradigms. Two regimes are compared:

1. Static training – the neural operator is trained offline on a pre‑computed dataset {(A_j, f_j, u*_j)}. Two families of loss functions are examined:

   • Error‑based loss (relative solution error) which directly penalizes the distance to the reference solution.
   • Residual‑based loss (relative PDE residual) which penalizes f − A Nθ(f).

   The authors also explore norm choices (ℓ₂, ℓ₁, H¹‑type) to weight different frequency components during training.

2. Dynamic (end‑to‑end) training – the full DL‑HIM cycle is unrolled for K steps inside a differentiable graph, and a cumulative loss over the K iterates is minimized. This exposes the neural operator to the evolving residual distribution encountered during inference, reducing the train‑inference mismatch inherent in static training.

Experiments on two representative DL‑HIMs—DeepONet‑based Hybrid Iterative Numerical Transferable Solver (HINTS) and FFT‑based Fourier Neural Solver (FNS)—show that the effectiveness of a loss function is problem‑dependent. For HINTS, a residual‑based ℓ₂ loss yields the most robust convergence, while for FNS an ℓ₁‑based error loss better suppresses high‑frequency artifacts. Dynamic training consistently improves convergence rates but incurs higher memory and compute costs.

The paper also scrutinizes acceleration strategies. Classical Anderson acceleration (AA) minimizes the norm of the update δ(k) = u(k+1) − u(k). When applied to DL‑HIMs with nonlinear neural operators, AA often fails to escape false fixed points because it still optimizes the wrong objective. To address this, the authors propose Physics‑Aware Anderson Acceleration (PA‑AA), which directly minimizes the physical residual norm ‖f − Au(k)‖. PA‑AA modifies the mixing coefficients and the least‑squares subproblem to operate in residual space rather than solution space.

Numerical results demonstrate that PA‑AA dramatically reduces the number of iterations required to achieve a residual below 10⁻⁸—typically a 30–40 % improvement over standard AA—and eliminates stagnation in cases where AA diverges or oscillates. When combined with dynamic training, PA‑AA yields the most reliable solver, even on high‑resolution grids (e.g., 4096 × 4096) and for non‑symmetric or indefinite matrices.

In conclusion, the reliability of DL‑HIMs does not hinge solely on the choice of neural architecture. It is critically governed by physically informed training objectives and update strategies that target the true PDE residual. By aligning the loss function with the residual propagation and employing physics‑aware acceleration, false fixed points can be avoided, leading to stable, fast convergence suitable for scientific computing. The work thus resolves ongoing controversies about AI‑based PDE solvers and provides clear guidelines for future development: prioritize residual‑based or hybrid loss designs, consider dynamic end‑to‑end training when resources allow, and replace standard AA with PA‑AA for any nonlinear neural operator component.