Comparing Measures of Sparsity

Sparsity of representations of signals has been shown to be a key concept of fundamental importance in fields such as blind source separation, compression, sampling and signal analysis. The aim of this paper is to compare several commonlyused sparsity measures based on intuitive attributes. Intuitively, a sparse representation is one in which a small number of coefficients contain a large proportion of the energy. In this paper six properties are discussed: (Robin Hood, Scaling, Rising Tide, Cloning, Bill Gates and Babies), each of which a sparsity measure should have. The main contributions of this paper are the proofs and the associated summary table which classify commonly-used sparsity measures based on whether or not they satisfy these six propositions and the corresponding proofs. Only one of these measures satisfies all six: The Gini Index. measures based on whether or not they satisfy these six propositions and the corresponding proofs. Only one of these measures satisfies all six: The Gini Index.

💡 Research Summary

The paper “Comparing Measures of Sparsity” conducts a systematic evaluation of fifteen widely‑used sparsity metrics against six intuitive criteria derived from the notion of wealth inequality. The authors first formalize sparsity as a situation where a small subset of coefficients carries most of the signal energy, and they translate this intuition into six mathematical properties:

1. Robin Hood (D1) – transferring a small amount from a larger coefficient to a smaller one must reduce the sparsity measure.
2. Scaling (D2) – multiplying all coefficients by a positive constant must leave the sparsity value unchanged.
3. Rising Tide (D3) – adding a positive constant to every coefficient must decrease sparsity (except when all coefficients are already equal).
4. Cloning (D4) – concatenating an exact copy of a vector (or multiple copies) must not alter the sparsity value.
5. Bill Gates (P1) – making a single coefficient arbitrarily large should drive the sparsity measure to its maximum.
6. Babies (P2) – appending zero‑valued coefficients should increase sparsity.

These six properties capture the extremes of the sparsity spectrum: the most sparse case (one coefficient dominates) and the least sparse case (all coefficients equal). The authors prove two theorems that reveal logical dependencies among the criteria:

• Theorem 2.1 shows that any measure satisfying D1 and D2 automatically satisfies P1.
• Theorem 2.2 demonstrates that a measure satisfying D1, D2, and D4 necessarily satisfies P2.

The fifteen candidate measures include the ℓ₀ “count‑of‑non‑zeros”, the ℓₚ family for 0 < p ≤ 1, the ℓ₁ norm, tanh‑based functions, logarithmic penalties, kurtosis (κ₄), the uθ range‑percentage metric, several entropy‑based diversities (Shannon, modified Shannon, Gaussian), the Hoyer normalized sparsity, and the Gini Index. For each metric the authors either adapt the sign or apply a monotonic transformation so that larger numerical values correspond to higher sparsity, facilitating a uniform comparison.

A detailed table (Table I) records whether each metric satisfies each of the six criteria. The analysis reveals that most metrics violate at least one property. For instance, ℓ₀ is not scale‑invariant (fails D2); ℓ₁ changes under cloning (fails D4); entropy‑based measures increase when zero entries are added (violating P2); and many ℓₚ norms with p < 1 do not fully respect the Robin Hood principle because they overweight large coefficients.

The Gini Index stands out as the sole metric that fulfills all six criteria. Defined for ordered coefficients (c_{(1)}\le …\le c_{(N)}) as
\