On the rate distortion function of Bernoulli Gaussian sequences

In this paper, we study the rate distortion function of the i.i.d sequence of multiplications of a Bernoulli $p$ random variable and a gaussian random variable $\sim N(0,1)$. We use a new technique in the derivation of the lower bound in which we establish the duality between channel coding and lossy source coding in the strong sense. We improve the lower bound on the rate distortion function over the best known lower bound by $p\log_2\frac{1}{p}$ if distortion $D$ is small. This has some interesting implications on sparse signals where $p$ is small since the known gap between the lower and upper bound is $H(p)$. This improvement in the lower bound shows that the lower and upper bounds are almost identical for sparse signals with small distortion because $\lim\limits_{p\to 0}\frac{p\log_2\frac{1}{p}}{H(p)}=1$.

💡 Research Summary

The paper investigates the rate‑distortion function R(D) of an i.i.d. sequence formed by multiplying a Bernoulli(p) random variable with a standard Gaussian N(0,1). This “Bernoulli‑Gaussian” source, denoted Ξ(p,σ²) with σ²=1, is a mixture of a discrete (the Bernoulli mask) and a continuous (the Gaussian amplitude) component, making its rate‑distortion analysis non‑trivial.

Background and known bounds
The authors first recall the classical rate‑distortion theory for both discrete and continuous sources, emphasizing that the Shannon lower bound h(X) – (1/2)log(2πeD) does not apply directly because the differential entropy of Ξ(p,1) is −∞. Existing literature provides three simple bounds:

1. Upper bound 1: H(p) + p·R(Dp,N(0,1)) = H(p) + p·½log(1/D) (for D ≤ 1). This corresponds to losslessly coding the mask bⁿ and then applying a Gaussian coder to the non‑zero entries.
2. Upper bound 2: R(D,N(0,p)) = ½log(p/D) (for D ≤ p), obtained by treating the whole source as a Gaussian with variance p.
3. Lower bound: p·R(Dp,N(0,1)) = p·½log(1/D), which assumes the decoder already knows the mask bⁿ for free. The gap between the best upper bound (≈H(p)) and this lower bound can be as large as H(p), especially when p is small (sparse signals).

New lower‑bound technique
To tighten the lower bound, the authors exploit a strong duality between channel coding and lossy source coding. They decompose the mutual information between the encoder output aⁿ_R (a binary string of length nR) and the source (bⁿ, sⁿ) into two parts:

• I(aⁿ_R; bⁿ), the information needed to convey the Bernoulli mask.
• I(aⁿ_R; sⁿ | bⁿ), the information needed to describe the Gaussian amplitudes given the mask.

Lemma 2 shows that nR ≥ I(aⁿ_R; bⁿ) + I(aⁿ_R; sⁿ | bⁿ). The second term is lower‑bounded exactly as in the previous simple lower bound, yielding p·R(D,N(0,p)). The novelty lies in bounding the first term. By interpreting the source‑coding system as a “lossy coding channel” whose capacity must be at least I(aⁿ_R; bⁿ), the authors apply random‑coding arguments, typical‑set analysis, and the data‑processing inequality to derive a non‑trivial lower bound on I(aⁿ_R; bⁿ).

The resulting expression involves three auxiliary parameters:

• L ≥ 0, the number of bits allocated per mask bit;
• U ≥ L, a threshold on the absolute value of the Gaussian component;
• r ∈