A Constructive Method to Maximize Entropy under Marginal Constraints

We study the problem of maximizing R{é}nyi entropy of order $2$ (equivalently, minimizing the index of coincidence) over the set of joint distributions with prescribed marginals. A closed-form optimizer is known under a feasibility condition on the marginals; we show that this condition is highly restrictive. We then provide an explicit construction of an optimal coupling for arbitrary marginals. Our approach characterizes the optimizer’s structure and yields an iterative algorithm that terminates in finite time, returning an exact solution after at most $p-1$ updates, where $p$ is the number of rows.

💡 Research Summary

The paper addresses the problem of maximizing Rényi entropy of order 2 (equivalently minimizing the index of coincidence, IC) for a joint distribution π over discrete variables X∈{1,…,p} and Y∈{1,…,q} subject to fixed marginal distributions μ∈Sₚ and ν∈S_q. This problem, termed “Maximum Rényi Entropy / Minimum Index of Coincidence,” is relevant in cryptanalysis, statistical physics, and information theory because a high‑order Rényi entropy corresponds to a low probability of accidental coincidences.

Background and limitation of existing results.
When the Rényi order is α=1 (Shannon entropy), the product coupling π×=μ⊗ν maximizes entropy. For α=2, a closed‑form optimizer π⁺ has been known only under the feasibility condition

  p μ₁ + q ν₁ ≥ 1  (3)

where μ₁ and ν₁ are the smallest marginal probabilities. Under (3) the “additive” coupling

 π⁺_{u,v}= (μ_u q + ν_v p − 1)/(pq)

satisfies all marginal constraints and is non‑negative, thus optimal. The authors first demonstrate that condition (3) is extremely restrictive: for generic random marginals the inequality fails with high probability, especially as p and q grow. Consequently, the known solution is of limited practical use.

KKT‑based structural analysis.
The authors formulate the minimization of IC(π)=∑π_{u,v}² as a convex program with linear equality constraints (the marginals) and non‑negativity constraints. By invoking Slater’s condition they guarantee that any optimal solution satisfies the Karush‑Kuhn‑Tucker (KKT) conditions. Introducing Lagrange multipliers λ_u, ω_v (for the marginal constraints) and r_{u,v} (for non‑negativity), the optimality condition can be written as

 π*{u,v}=π⁺{u,v} − R_{u,·}/q − R_{·,v}/p + R/(pq),

where R_{u,·}=∑v r{u,v}, R_{·,v}=∑u r{u,v}, R=∑{u,v} r{u,v}, and r_{u,v}≥0. This expression (Proposition 5) shows that the deviation from the additive coupling is entirely governed by the non‑negativity multipliers.

Monotonicity and staircase of zeros.
Assuming the marginals are ordered increasingly (μ₁≤…≤μ_p, ν₁≤…≤ν_q), the authors prove (Proposition 6) that any optimal π* is monotone in both row and column indices. Consequently, zero entries must occupy a “staircase” region: each row begins with a block of zeros, the length of which is non‑increasing across rows, and similarly for columns. They formalize the last zero in row u as q_u and in column v as p_v (Definition 3). This structural insight reduces the problem of determining π* to identifying the staircase corners and the multiplier r.

Rectangular milestone and closed‑form correction.
As a tractable intermediate case, the paper studies the situation where the zero set forms a single upper‑left rectangle of size p₁ × q₁. Under this assumption (Equation 7) the optimal coupling takes the simple form

 π*{u,v}=0 if u≤p₁ and v≤q₁,
 π*{u,v}=π⁺{u,v} − R{u,·}/q − R_{·,v}/p + R/(pq) otherwise.

The authors define the “mass loss” Δ caused by truncating π⁺ on the rectangle (Equations 9‑11) and derive explicit formulas for the correction terms R_{u,·}, R_{·,v}, R in terms of the marginals and Δ. This yields a closed‑form “rectangle‑solution” \tildeπ⁺, which exactly satisfies the marginal constraints while eliminating the rectangular block of zeros.

Iterative algorithm for the general staircase.
When the zero pattern is not a single rectangle but a staircase, the authors iteratively apply the rectangle‑solution transformation. At each iteration they locate the top‑most left‑most zero block (the current rectangle), compute the corresponding correction R using the mass‑loss formulas, replace the entries inside the block by zero, and redistribute the removed mass according to the additive coupling on the remaining support. This operation strictly reduces the number of rows containing zeros by at least one. Repeating at most p−1 times eliminates all zeros, producing the exact optimal coupling π*. The algorithm terminates in finite time, requires only elementary arithmetic operations, and runs in O(pq) time per iteration, yielding an overall polynomial‑time procedure.

Empirical assessment of condition (3).
Section 6 presents Monte‑Carlo experiments showing that for random marginals the feasibility condition p μ₁ + q ν₁ ≥ 1 holds with probability decreasing rapidly as p and q increase (e.g., <5 % for p=q=50). This confirms that the previously known closed‑form solution applies only in a narrow corner of the parameter space, underscoring the practical relevance of the new construction.

Conclusions and outlook.
The paper makes three principal contributions: (i) it quantifies the rarity of the classical feasibility condition, (ii) it uncovers the monotone staircase structure of optimal couplings for Rényi‑2 entropy under marginal constraints, and (iii) it provides a constructive, finite‑step algorithm that yields the exact optimizer for arbitrary marginals. The methodology is readily extensible to continuous alphabets, other Rényi orders, and multi‑dimensional coupling problems, opening avenues for future research in information‑theoretic security, optimal transport, and statistical inference.