Bayesian Inference for Discrete Markov Random Fields Through Coordinate Rescaling

Discrete Markov random fields are undirected graphical models that capture complex conditional dependencies between discrete variables. Conducting exact posterior inference in these models is often computationally challenging because evaluating their normalizing constant requires summation over all possible state configurations, and the size of this state space grows exponentially with the number of variables and their possible states. As a result, exact likelihood-based inference is infeasible in many practical settings, and existing methods, such as Double Metropolis-Hastings or pseudo-likelihood approximations, either scale poorly to large systems or underestimate posterior variability. To address these limitations, we propose a new class of coordinate-rescaling sampling methods that transform pseudo-likelihood-based posteriors toward the target posterior while preserving computational efficiency. The resulting samplers retain scalability while improving uncertainty quantification. In simulation studies, we compare the proposed methods to existing approaches and demonstrate that coordinate-rescaling sampling yields more accurate estimates of posterior variability, providing a scalable and reliable approach to Bayesian inference in discrete MRFs.

💡 Research Summary

The paper tackles the long‑standing computational bottleneck in Bayesian inference for discrete Markov random fields (MRFs), namely the intractable normalizing constant that appears both in the likelihood and the posterior (double intractability). While exact inference is feasible only for very small graphs, two main approximate strategies dominate the literature. The Double Metropolis‑Hastings (DMH) algorithm approximates the exchange sampler by embedding an inner MCMC loop that generates auxiliary data; this cancels the normalizing constants but incurs a heavy computational cost because each Metropolis proposal requires a full inner chain. In contrast, pseudo‑likelihood (PL) replaces the full likelihood with a product of conditional distributions, eliminating the need to compute the normalizing constant altogether. PL is extremely fast and yields unbiased point estimates, yet it systematically underestimates posterior variance, leading to over‑confident inference.

The authors propose a novel class of samplers called Coordinate‑Rescaling (CoRe). The key idea is to embed a linear transformation directly into the MCMC sampling scheme, mapping the PL‑based parameter space (η) to a rescaled space (β) via β = A(η − η★) + η★, where η★ is a point estimate (e.g., MAP or MPLE) and A is a fixed rescaling matrix designed to approximate the covariance structure of the true posterior. Because the Jacobian of this transformation is constant (det A⁻¹), the Metropolis‑Hastings acceptance ratio simplifies to a ratio of PL densities and priors evaluated at the transformed parameters, completely avoiding the intractable Z(η). The algorithm proceeds by proposing β′ from a convenient proposal distribution q(·|β), mapping back to η′, and accepting/rejecting using the PL‑based acceptance probability. By construction, the transformed posterior π(β|X) inherits the shape of the target posterior, improving exploration especially in high‑dimensional, sharply peaked distributions.

Theoretical discussion highlights that if A accurately captures the target covariance, the β‑space becomes nearly isotropic, leading to higher acceptance rates and faster mixing. The method retains the scalability of PL because each iteration requires only evaluation of conditional probabilities, yet it restores realistic posterior uncertainty because the rescaling inflates the variance to match that of the full‑likelihood posterior.

Empirical evaluation includes four competing methods: (i) DMH, (ii) plain PL, (iii) post‑hoc calibration of PL draws (Bouranis et al., 2017), and (iv) an empirical‑likelihood‑based sampler. Simulations focus on ordinal MRFs (OMRFs) with 10–30 variables and 2–4 categories per variable, a setting where the normalizing constant is astronomically large. Results show that CoRe achieves point‑estimate accuracy comparable to all methods, posterior variance estimates virtually indistinguishable from DMH (and far superior to PL and post‑hoc calibration), and a runtime of roughly 28 seconds on a modern laptop—about 20× faster than DMH (≈9 minutes) and only modestly slower than plain PL (≈14 seconds). The authors also provide visualizations (Figure 1) illustrating how CoRe’s posterior density aligns with the exact posterior while preserving computational tractability.

Limitations are acknowledged. The rescaling matrix A is derived from the MPLE, which may be a coarse approximation of the true posterior covariance in highly connected or heterogeneous graphs. The current development is restricted to ordinal variables; extending the approach to non‑ordinal categorical MRFs may require more sophisticated transformations. Future work is suggested on adaptive estimation of A during sampling, theoretical convergence guarantees for high‑dimensional settings, and broader applicability to other discrete graphical models.

In summary, the paper introduces a practical, theoretically motivated sampling framework that bridges the gap between fast but variance‑deficient PL methods and accurate but computationally heavy DMH. By embedding a coordinate‑rescaling step within the Metropolis‑Hastings algorithm, the authors deliver a scalable solution that yields reliable uncertainty quantification for Bayesian inference in discrete MRFs, opening the door to applying fully Bayesian analysis to much larger and more complex discrete graphical models than previously feasible.