HQP: Sensitivity-Aware Hybrid Quantization and Pruning for Ultra-Low-Latency Edge AI Inference

The escalating demand for high-fidelity, real-time inference in distributed edge-cloud environments necessitates aggressive model optimization to counteract severe latency and energy constraints. This paper introduces the Hybrid Quantization and Pruning (HQP) framework, a novel, integrated methodology designed to achieve synergistic model acceleration while adhering to strict quality guarantees. We detail a sensitivity-aware structural pruning algorithm that employs a dynamic weight sensitivity metric, derived from a highly efficient approximation of the Fisher Information Matrix (FIM), to guide the iterative removal of redundant filters. This pruning is strictly conditional, enforcing an adherence to a maximum permissible accuracy drop (Delta ax) before the model proceeds to 8-bit post-training quantization. This rigorous coordination is critical, as it ensures the resultant sparse model structure is maximally robust to quantization error and hardware-specific kernel optimization. Exhaustive evaluation across heterogeneous NVIDIA Jetson edge platforms, utilizing resource-efficient architectures like MobileNetV3 and ResNet-18, demonstrates that the HQP framework achieves a peak performance gain of 3.12 times inference speedup and a 55 percent model size reduction, while rigorously containing the accuracy drop below the 1.5 percent constraint. A comprehensive comparative analysis against conventional single-objective compression techniques validates the HQP framework as a superior, hardware-agnostic solution for deploying ultra-low-latency AI in resource-limited edge infrastructures.

💡 Research Summary

The paper introduces HQP (Hybrid Quantization and Pruning), a coordinated two‑stage compression framework designed for ultra‑low‑latency inference on resource‑constrained edge devices. The authors first identify a fundamental flaw in conventional pipelines: pruning and quantization are applied sequentially and independently, which often leads to outlier weights after pruning that inflate the dynamic range of tensors. This, in turn, forces a coarse quantization scale during post‑training quantization (PTQ) and can cause catastrophic accuracy loss.
HQP resolves this by (1) computing a filter‑level sensitivity metric S using a highly efficient diagonal approximation of the Fisher Information Matrix (FIM). This metric captures the second‑order impact of each filter on the loss landscape and can be obtained with a single backward pass over a modest calibration set, making it computationally cheap. (2) Performing structural (channel‑wise) pruning in an iterative loop that is explicitly constrained by a user‑defined maximum allowable accuracy drop Δₐₓ (e.g., 1.5 %). At each iteration a small fraction δ of the lowest‑sensitivity filters is removed, the pruned model is immediately validated, and pruning stops as soon as the accuracy drop exceeds Δₐₓ. This guarantees that the final sparse model is the most compressed version that still respects the quality budget.
Because the pruning step eliminates high‑variance, redundant filters, the resulting weight distribution has a reduced dynamic range R, which yields a smaller quantization step size s for the subsequent INT8 PTQ. The authors employ NVIDIA TensorRT’s KL‑divergence calibration to determine per‑layer scaling factors, and the structurally sparse model benefits from TensorRT’s layer fusion, dead‑layer elimination, and kernel auto‑tuning, turning theoretical compression gains into real latency reductions.
Experiments are conducted on two heterogeneous NVIDIA Jetson platforms: the low‑power Jetson Nano (5‑10 W) and the more capable Jetson Xavier NX (10‑15 W). Two representative networks, MobileNetV3‑Small and ResNet‑18, are compressed using HQP and compared against three baselines: (i) FP32 baseline, (ii) quantization‑only (INT8 PTQ), and (iii) pruning‑only (50 % sparsity). On the Xavier NX, HQP achieves a 3.12× speed‑up, a 55 % reduction in model size, and only a 1.4 % top‑1 accuracy drop, comfortably staying within the Δₐₓ ≤ 1.5 % constraint. The pruning‑only baseline yields 1.35× speed‑up with a 1.8 % accuracy loss, while quantization‑only gives 1.58× speed‑up but a 1.2 % loss; HQP outperforms both simultaneously.
A computational‑complexity analysis shows that HQP’s total cost C_HQP = N_calib·C_grad + T_prune·N_val·C_inf is dominated by a few thousand forward passes, whereas Quantization‑Aware Training (QAT) requires many epochs over the full training set, making HQP orders of magnitude more efficient for production pipelines.
In summary, HQP delivers a hardware‑agnostic, production‑ready method that jointly optimizes structural sparsity and low‑precision quantization while guaranteeing a user‑specified accuracy budget. The framework’s reliance on a theoretically grounded sensitivity metric, its conditional pruning loop, and its seamless integration with TensorRT make it a compelling solution for deploying high‑fidelity AI at the edge where latency, power, and memory are at a premium. Future work may extend HQP to other accelerator families (ASICs, FPGAs) and explore automated selection of Δₐₓ based on application‑level QoS requirements.