A simple algorithm for output range analysis for deep neural networks

This paper presents a novel approach for the output range estimation problem in Deep Neural Networks (DNNs) by integrating a Simulated Annealing (SA) algorithm tailored to operate within constrained domains and ensure convergence towards global optima. The method effectively addresses the challenges posed by the lack of local geometric information and the high non-linearity inherent to DNNs, making it applicable to a wide variety of architectures, with a special focus on Residual Networks (ResNets) due to their practical importance. Unlike existing methods, our algorithm imposes minimal assumptions on the internal architecture of neural networks, thereby extending its usability to complex models. Theoretical analysis guarantees convergence, while extensive empirical evaluations-including optimization tests involving functions with multiple local minima-demonstrate the robustness of our algorithm in navigating non-convex response surfaces. The experimental results highlight the algorithm’s efficiency in accurately estimating DNN output ranges, even in scenarios characterized by high non-linearity and complex constraints. For reproducibility, Python codes and datasets used in the experiments are publicly available through our GitHub repository.

💡 Research Summary

The paper tackles the problem of estimating the output range (minimum and maximum possible outputs) of deep neural networks (DNNs) when the inputs are confined to a bounded domain. Instead of relying on gradient information, mixed‑integer linear programming, or interval arithmetic—approaches that typically require architectural assumptions and suffer from scalability issues—the authors treat the network as a black‑box function F(x) and apply a global optimization technique based on Simulated Annealing (SA).

Key contributions are:

1. Algorithm Design – The proposed SA variant combines four simple mechanisms: (a) symmetric random proposals around the current point to avoid directional bias; (b) cyclic reflection at the domain boundaries, so any proposal that would leave the feasible hyper‑cube is mirrored back, preventing artificial clustering near edges; (c) a Metropolis‑style acceptance probability exp((f(x_t)‑f(x′))/T) with a temperature T that follows a predefined cooling schedule; (d) persistent tracking of the best‑so‑far value, ensuring that progress is never lost.

2. Theoretical Guarantees – Under the mild assumptions that the input set Ω is compact and the network output function F is continuous, the authors prove that if the proposal distribution is symmetric and has full support over Ω, and if the temperature sequence T_k tends to zero while the number of iterations at each temperature grows sufficiently, the Markov chain induced by the algorithm converges in probability to the global optimum. This extends classic SA convergence results (van Laarhoven & Aarts, 1987; Nourani & Andresen, 1998) to the case with reflective boundary handling.

3. Empirical Evaluation – Experiments are conducted on several Residual Networks (ResNet‑18, ResNet‑34, ResNet‑50) with input domains defined as hyper‑cubes