SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compression

The rapid growth in the parameter scale of large language models (LLMs) has created a high demand for efficient compression techniques. As a hardware-agnostic and highly compatible technique, low-rank compression has been widely adopted. However, existing methods typically compress each layer independently by minimizing per-layer reconstruction error, overlooking a critical limitation: the reconstruction error propagates and accumulates through the network, which leads to amplified global deviations from the full-precision baseline. To address this, we propose Self-Adaptive Error Suppression SVD (SAES-SVD), a LLMs compression framework that jointly optimizes intra-layer reconstruction and inter-layer error compensation. SAES-SVD is composed of two novel components: (1) Cumulative Error-Aware Layer Compression (CEALC), which formulates the compression objective as a combination of local reconstruction and weighted cumulative error compensation. Based on it, we derive a closed-form low-rank solution relied on second-order activation statistics, which explicitly aligns each layer’s output with its full-precision counterpart to compensate for accumulated errors. (2) Adaptive Collaborative Error Suppression (ACES), which automatically adjusts the weighting coefficient to enhance the low-rank structure of the compression objective in CEALC. Specifically, the coefficient is optimized to maximize the ratio between the Frobenius norm of the compressed layer’s output and that of the compression objective under a fixed rank, thus ensuring that the rank budget is utilized effectively. Extensive experiments across multiple LLM architectures and tasks show that, without fine-tuning or mixed-rank strategies, SAES-SVD consistently improves post-compression performance.

💡 Research Summary

The paper addresses a critical shortcoming of existing low‑rank compression methods for large language models (LLMs). Traditional SVD‑based techniques treat each layer independently, minimizing a per‑layer reconstruction loss that only measures how well the compressed weight reproduces the original output on the current input. This ignores the fact that upstream compression distorts the input distribution of downstream layers, causing errors to accumulate and amplify as depth increases. Consequently, even though each layer may be optimally truncated, the final model output can diverge substantially from the full‑precision baseline.

To remedy this, the authors propose SAES‑SVD (Self‑Adaptive Error Suppression SVD), a framework that jointly optimizes intra‑layer reconstruction fidelity and inter‑layer error compensation. SAES‑SVD consists of two novel components:

1. Cumulative Error‑Aware Layer Compression (CEALC).
   CEALC augments the standard loss with a weighted alignment term that forces the compressed layer’s output to match the full‑precision reference output computed on the uncompressed upstream activations. Formally, for layer ℓ the objective is
   \