EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation

While post-training compression techniques effectively reduce the memory footprint, latency, and power consumption of Large Language Models (LLMs), they often result in noticeable accuracy degradation and remain limited by hardware and kernel constraints that restrict supported compression formats - ultimately reducing flexibility across a wide range of deployment scenarios. In this work, we propose EoRA - a novel $\textbf{fine-tuning-free}$ method that augments compressed LLMs with low-rank matrices, allowing users to rapidly enhance task-specific performance and freely balance the trade-off between accuracy and computational overhead beyond the constraints of compression formats. EoRA consistently outperforms prior fine-tuning-free low rank methods in recovering the accuracy of compressed LLMs, achieving notable accuracy improvements (e.g., $\mathbf{10.84%}$ on ARC-Challenge, $\mathbf{6.74%}$ on MathQA, and $\mathbf{11.45%}$ on GSM8K for LLaMA3-8B compressed to 3-bit). We also introduce an optimized CUDA kernel, accelerating inference by up to 1.4x and reducing memory overhead through quantizing EoRA. Overall, EoRA offers a prompt solution for improving the accuracy of compressed models under varying user requirements, enabling more efficient and flexible deployment of LLMs. Code is available at https://github.com/NVlabs/EoRA.

💡 Research Summary

The paper introduces EoRA (Eigen‑space Low‑Rank Approximation), a fine‑tuning‑free technique designed to restore or even improve the task‑specific performance of large language models (LLMs) that have been compressed through post‑training quantization, pruning, or a combination of both. The authors first identify a practical limitation of existing compression pipelines: while they reduce memory, latency, and power consumption, they often cause a noticeable drop in accuracy and are constrained by hardware‑specific formats (e.g., 2:4 sparsity, integer‑only kernels). Consequently, users cannot freely trade off accuracy against computational overhead for diverse deployment scenarios.

EoRA reframes this situation as a “customized compensation” problem: given a compressed backbone whose weights remain unchanged, attach lightweight, low‑rank residual modules that compensate for the compression error on a per‑task basis. The key novelty lies in how the compensation matrices are computed without any gradient‑based training. The method proceeds in three steps:

1. Eigenspace Projection – For each linear layer, the average activation matrix (\tilde{X}) over a small calibration set (typically 64–128 samples) is collected. The covariance (\tilde{X}\tilde{X}^\top) is eigendecomposed as (Q\Lambda Q^\top). The eigenvalues (\Lambda) serve as importance scores for each activation channel, indicating how much each direction contributes to the downstream task.

2. Error Projection and Low‑Rank Approximation – The compression error (\Delta W = W - \hat{W}) (difference between the original full‑precision weight and its compressed counterpart) is projected into the eigenspace using the transformation (Q’ = Q\sqrt{\Lambda}), yielding (\Delta W’ = \Delta W Q’). A rank‑(r) singular value decomposition (SVD) is then applied to (\Delta W’), producing (U’\Sigma’V’^\top). The low‑rank factors are defined as (B’ = U’\Sigma’) and (A’ = V’^\top).

3. Back‑Projection to Original Space – The factor (A’) is multiplied by the inverse transformation (Q’^{-1} = \sqrt{\Lambda}^{-1} Q^\top) to obtain the final compensation matrices (A = A’ Q’^{-1}) and (B = B’). The forward pass of a compressed linear layer becomes (\hat{W}X + BA X).

The authors prove (Theorem 1) that minimizing the Frobenius norm of the projected error (|\Delta W’ - B’A’|_F) via SVD is mathematically equivalent to minimizing the original layer‑wise compression loss (|\Delta W X - BA X|_F). This guarantees that the low‑rank approximation is optimal with respect to the task‑specific activation distribution, unlike naïve SVD on (\Delta W) which ignores data statistics.

To keep inference overhead low, the authors design a fused CUDA kernel that simultaneously performs the low‑rank matrix multiplication and the quantization step required by the compressed backbone. This kernel reduces memory traffic and achieves up to 1.4× speed‑up on an NVIDIA H100 GPU compared with a naïve implementation. The low‑rank matrices themselves are quantized (e.g., to 4‑bit) without significant loss, further shrinking the memory footprint (additional 0.2–0.4 GB for an 8‑B model).

Experimental Evaluation – The method is evaluated on LLaMA‑2 (7 B/13 B) and LLaMA‑3 (8 B) models compressed with SparseGPT (2:4 structured sparsity) and GPTQ (3‑bit quantization). Baselines include ZeroQuant‑V2 (simple SVD on (\Delta W)), Act‑S (activation‑scaled SVD), and ApiQ (gradient‑based low‑rank adaptation). Results show that EoRA consistently outperforms all baselines, especially for aggressively compressed models. For LLaMA‑3‑8 B quantized to 3‑bit and pruned to 2:4 sparsity, EoRA improves ARC‑Challenge by 10.84 %, MathQA by 6.74 %, and GSM8K by 11.45 % absolute accuracy—gains of 2–3× over ZeroQuant‑V2. Even when only sparsity is applied, EoRA adds 4.53 %–11.83 % accuracy across the three benchmarks.

The authors also demonstrate that EoRA can serve as an excellent initialization for LoRA‑style fine‑tuning: a short LoRA training phase on top of EoRA‑augmented models yields higher final performance than training LoRA from scratch.

Implications – EoRA offers a practical, deployment‑ready solution: a single compressed backbone can be shipped, and task‑specific low‑rank adapters can be loaded on demand, enabling flexible accuracy‑latency trade‑offs without re‑compressing the model or performing expensive fine‑tuning. The method respects hardware constraints (since the backbone remains unchanged) while providing a data‑driven, mathematically optimal compensation mechanism.

In summary, EoRA introduces a theoretically grounded, calibration‑driven low‑rank compensation framework that restores and often surpasses the original accuracy of compressed LLMs, delivers measurable inference speed‑ups through a custom kernel, and integrates seamlessly with existing multi‑adapter inference stacks, thereby advancing the state of the art in efficient, flexible LLM deployment.