Revisit, Extend, and Enhance Hessian-Free Influence Functions

Influence functions serve as crucial tools for assessing sample influence in model interpretation, subset training set selection, noisy label detection, and more. By employing the first-order Taylor extension, influence functions can estimate sample influence without the need for expensive model retraining. However, applying influence functions directly to deep models presents challenges, primarily due to the non-convex nature of the loss function and the large size of model parameters. This difficulty not only makes computing the inverse of the Hessian matrix costly but also renders it non-existent in some cases. Various approaches, including matrix decomposition, have been explored to expedite and approximate the inversion of the Hessian matrix, with the aim of making influence functions applicable to deep models. In this paper, we revisit a specific, albeit naive, yet effective approximation method known as TracIn. This method substitutes the inverse of the Hessian matrix with an identity matrix. We provide deeper insights into why this simple approximation method performs well. Furthermore, we extend its applications beyond measuring model utility to include considerations of fairness and robustness. Finally, we enhance TracIn through an ensemble strategy. To validate its effectiveness, we conduct experiments on synthetic data and extensive evaluations on noisy label detection, sample selection for large language model fine-tuning, and defense against adversarial attacks.

💡 Research Summary

Influence functions have become a cornerstone of data‑centric machine learning, enabling practitioners to assess how individual training samples affect a model’s performance without costly retraining. The classic formulation relies on a first‑order Taylor expansion that involves the inverse Hessian of the training loss, (\nabla_{\theta}\ell(V;\hat\theta)^{\top}H_{\hat\theta}^{-1}\nabla_{\theta}\ell(z_i;\hat\theta)). While theoretically sound for convex problems, this expression becomes impractical for modern deep networks: the loss is non‑convex, the parameter space can contain millions of dimensions, and the Hessian is often ill‑conditioned or even indefinite, making its inverse either computationally prohibitive or mathematically undefined.

A line of prior work (LiSSA, Kronecker‑Factored Approximation, Arnoldi, etc.) attempts to approximate (H^{-1}) through sophisticated matrix‑free techniques, but these methods still incur substantial overhead and can be unstable. In contrast, the TracIn family—referred to in this paper as the Inner Product (IP) method—takes a dramatically simpler route: it replaces the inverse Hessian with the identity matrix, reducing the influence estimate to the inner product between the gradient of the validation set and the gradient of a single training point. Formally, \