Expressiveness of Multi-Neuron Convex Relaxations in Neural Network Certification

Neural network certification methods heavily rely on convex relaxations to provide robustness guarantees. However, these relaxations are often imprecise: even the most accurate single-neuron relaxation is incomplete for general ReLU networks, a limitation known as the single-neuron convex barrier. While multi-neuron relaxations have been heuristically applied to address this issue, two central questions arise: (i) whether they overcome the convex barrier, and if not, (ii) whether they offer theoretical capabilities beyond those of single-neuron relaxations. In this work, we present the first rigorous analysis of the expressiveness of multi-neuron relaxations. Perhaps surprisingly, we show that they are inherently incomplete, even when allocated sufficient resources to capture finitely many neurons and layers optimally. This result extends the single-neuron barrier to a universal convex barrier for neural network certification. On the positive side, we show that completeness can be achieved by either (i) augmenting the network with a polynomial number of carefully designed ReLU neurons or (ii) partitioning the input domain into convex sub-polytopes, thereby distinguishing multi-neuron relaxations from single-neuron ones which are unable to realize the former and have worse partition complexity for the latter. Our findings establish a foundation for multi-neuron relaxations and point to new directions for certified robustness, including training methods tailored to multi-neuron relaxations and verification methods with multi-neuron relaxations as the main subroutine.

💡 Research Summary

This paper provides the first rigorous theoretical investigation of multi‑neuron convex relaxations for certifying ReLU neural networks. Existing certification methods rely on convex relaxations to over‑approximate the reachable output set, but even the most precise single‑neuron relaxation (the Triangle relaxation) suffers from the “single‑neuron convex barrier”: it cannot yield exact bounds for general ReLU networks. The authors ask whether moving from single‑neuron to multi‑neuron relaxations can break this barrier, and if not, whether multi‑neuron relaxations still offer any fundamental advantage.

The authors formalize layerwise multi‑neuron relaxations (denoted P₁) and the more powerful cross‑r‑layer relaxations (Pᵣ, which compute the convex hull of r consecutive layers). They prove a universal incompleteness result: for any ReLU network, any layerwise multi‑neuron relaxation—no matter how many neurons are jointly considered in each layer—fails to provide exact lower (or upper) bounds on some inputs. The proof hinges on two observations. First, if an intermediate representation U of the network is non‑convex, its convex hull conv(U) contains points that are not reachable by the actual network. Second (Lemma 3.1), constraints generated at a given hidden layer cannot prune points that belong to conv(U) but lie outside U, because layerwise relaxations never introduce cross‑layer affine constraints. Consequently, the final relaxation may evaluate the network on an infeasible point in conv(U), producing a bound that is arbitrarily loose. This establishes a “universal convex barrier” that extends the single‑neuron barrier to all convex relaxations, even those that consider arbitrarily many neurons or layers, and also to networks with non‑polynomial activations such as tanh.

Despite this negative result, the paper identifies two constructive ways to achieve completeness. (i) By augmenting the original network with a polynomial‑size set of carefully designed ReLU neurons, any continuous piecewise‑linear function can be encoded in a network that is exactly bounded by some layerwise multi‑neuron relaxation (Theorem 5.1). This shows that multi‑neuron relaxations preserve the expressive power of ReLU networks, a property impossible for any single‑neuron relaxation. (ii) By partitioning the input domain into a modest number of convex sub‑polytopes, one can apply a layerwise relaxation on each sub‑region and obtain exact bounds globally. The required partition complexity for multi‑neuron relaxations is provably lower than that needed for single‑neuron relaxations, highlighting a practical efficiency gain.

The authors illustrate these ideas with a case study on the two‑dimensional max function. While single‑neuron relaxations cannot certify it, a modest‑k multi‑neuron relaxation (Mₖ) captures the exact output, confirming the theoretical advantage. Finally, the paper discusses practical implications: (a) training objectives that encourage network structures amenable to multi‑neuron relaxations, (b) verification pipelines that treat multi‑neuron relaxations as the core subroutine, and (c) potential extensions to mixed‑integer programming and branch‑and‑bound methods.

In summary, the work proves that convex relaxations—whether single‑ or multi‑neuron—are fundamentally incomplete for general ReLU networks, establishing a universal convex barrier. Nevertheless, by enriching the network architecture or by intelligently partitioning the input space, multi‑neuron relaxations can achieve exact certification where single‑neuron methods cannot, offering both theoretical insight and practical pathways for future certified‑robustness research.