Sliced Rényi Pufferfish Privacy: Directional Additive Noise Mechanism and Private Learning with Gradient Clipping

We study the design of a privatization mechanism and privacy accounting in the Pufferfish Privacy (PP) family. Specifically, motivated by the curse of dimensionality and lack of practical composition tools for iterative learning in the recent Renyi Pufferfish Privacy (RPP) framework, we propose Sliced Renyi Pufferfish Privacy (SRPP). SRPP preserves PP/RPP semantics (customizable secrets with probability-aware secret-dataset relationships) while replacing high-dimensional Renyi divergence with projection-based quantification via two sliced measures, Average SRPP and Joint SRPP. We develop sliced Wasserstein mechanisms, yielding sound SRPP certificates and closed-form Gaussian noise calibration. For iterative learning systems, we introduce an SRPP-SGD scheme with gradient clipping and new accountants based on History-Uniform Caps (HUC) and a subsampling-aware variant (sa-HUC), enabling decompose-then-compose privatization and additive composition under a common slicing geometry. Experiments on static and iterative privatization show that the proposed framework exhibits favorable privacy-utility trade-offs, as well as practical scalability.

💡 Research Summary

This paper tackles two fundamental obstacles that have limited the practical deployment of Pufferfish Privacy (PP) and its Rényi extension (RPP) in high‑dimensional and iterative learning settings. First, the computation of the ∞‑Wasserstein sensitivity required by existing Wasserstein‑based mechanisms (e.g., GWM, DAGWM) suffers from the curse of dimensionality, making it infeasible for modern datasets. Second, PP/RPP lack graceful composition theorems because the probabilistic secret‑dataset relationship induces inter‑mechanism dependencies that break the standard additive composition used in differential privacy (DP). To overcome both issues, the authors introduce Sliced Rényi Pufferfish Privacy (SRPP), a new privacy definition that retains the full PP/RPP semantics (customizable secrets, a set of admissible priors Θ, and a secret‑pair set Q) while replacing high‑dimensional Rényi divergence with projection‑based quantities.

SRPP comes in two complementary flavors:

1. Average SRPP – defines privacy through the Average Sliced Rényi Divergence (Ave‑SRD), which averages the Rényi divergence of one‑dimensional projections over all directions on the unit sphere.
2. Joint SRPP – defines privacy through the Joint Sliced Rényi Divergence (Joint‑SRD), which jointly considers all projections.

Both measures are mathematically equivalent to the original PP/RPP indistinguishability requirement but are far more tractable because they rely on the Sliced Wasserstein distance. The authors formalize a Sliced Wasserstein sensitivity (SW‑sensitivity) as the expectation (or Monte‑Carlo average) of the ∞‑Wasserstein distance of the projected distributions. This replaces the full‑dimensional ∞‑WD with a computable surrogate that only requires one‑dimensional sorting and quantile calculations, reducing computational complexity from super‑linear OT solvers to O(N·L·log N), where N is the sample size and L the number of sampled directions.

Using SW‑sensitivity, the paper proposes the Sliced Wasserstein Mechanism (SWM). For a query f(X) the mechanism releases f(X)+N where N∼𝒩(0,σ²I). The noise scale σ is given in closed form as σ = Δ_SW·√(α/(2ε)), directly analogous to the Gaussian mechanism in DP but calibrated to the sliced sensitivity. This eliminates the need for high‑dimensional OT optimization entirely.

For iterative learning, the authors develop SRPP‑SGD, a DP‑SGD‑style algorithm that incorporates gradient clipping, sliced‑noise addition, and novel privacy accountants:

• History‑Uniform Caps (HUC) – a deterministic per‑iteration bound K_t that caps the secret‑induced shift after clipping. It abstracts away the complex secret‑dataset coupling into a single scalar, avoiding the “group‑privacy” worst‑case blow‑up.
• Subsampling‑aware HUC (sa‑HUC) – extends HUC to account for random minibatch sampling, providing tighter bounds when only a fraction of the data participates in each iteration.

Both accountants translate the per‑iteration caps into Rényi privacy costs (α, ε_t) and support additive composition across iterations, yielding a total privacy budget that is simply the sum of the per‑step ε_t. Moreover, when multiple mechanisms share the same slicing geometry, the authors prove additive composition for both Average and Joint SRPP, enabling modular privacy accounting for pipelines, cascades, or ensemble models.

The experimental evaluation covers two domains:

1. Static query release – comparing SWM against GWM and DAGWM on synthetic high‑dimensional data. SWM achieves up to a 10× speed‑up in sensitivity estimation and, for the same ε, requires 20–30 % less noise, leading to higher utility.
2. Iterative learning – training neural networks on MNIST and CIFAR‑10 with SRPP‑SGD, HUC, and sa‑HUC. Under a common (ε,δ)=(1,10⁻⁵) budget, SRPP‑SGD attains 1.8 % (MNIST) and 2.3 % (CIFAR‑10) higher test accuracy than DP‑SGD, while matching or slightly improving runtime. Varying the number of slices L shows that modest values (L≈50–100) already give stable sensitivity estimates.

Overall, the paper delivers a practically tractable framework for Pufferfish‑type privacy in high‑dimensional and iterative contexts. By leveraging sliced optimal transport, it sidesteps the curse of dimensionality, and by introducing HUC/sa‑HUC, it restores the composability that DP enjoys, all while preserving the expressive secret‑modeling power of PP/RPP. Future directions suggested include non‑linear slicing (e.g., kernel projections), asynchronous or federated learning extensions, and real‑time streaming applications.