IPF for Discrete Chain Factor Graphs

Iterative Proportional Fitting (IPF), combined with EM, is commonly used as an algorithm for likelihood maximization in undirected graphical models. In this paper, we present two iterative algorithms that generalize upon IPF. The first one is for likelihood maximization in discrete chain factor graphs, which we define as a wide class of discrete variable models including undirected graphical models and Bayesian networks, but also chain graphs and sigmoid belief networks. The second one is for conditional likelihood maximization in standard undirected models and Bayesian networks. In both algorithms, the iteration steps are expressed in closed form. Numerical simulations show that the algorithms are competitive with state of the art methods.

💡 Research Summary

**
The paper presents two novel iterative algorithms that extend the classic Iterative Proportional Fitting (IPF) method, often combined with Expectation‑Maximization (EM), to a broader class of discrete probabilistic models called Discrete Chain Factor Graphs (DCFGs). DCFGs unify a wide range of existing models—including undirected graphical models, Bayesian networks, chain graphs, and sigmoid belief networks—under a single framework where all variables are discrete but the dependency structure can be highly complex.

The first algorithm addresses maximum likelihood estimation for DCFGs. It follows an EM‑like two‑step procedure: an E‑step that computes expected sufficient statistics for hidden or latent variables using the current parameter estimates, and an M‑step that updates the parameters by a proportional scaling operation. This scaling generalizes the traditional IPF update rule, ensuring that each factor’s parameters are adjusted so that the overall joint distribution matches the observed sufficient statistics while respecting the conditional independence constraints inherent in the factor graph. The update formulas are derived in closed form, which eliminates the need for numerical optimization within each iteration.

The second algorithm focuses on conditional likelihood maximization for standard undirected models and Bayesian networks. Recognizing that conventional EM can be inefficient when the objective is a conditional likelihood, the authors introduce Conditional IPF. This method updates parameters using only the sufficient statistics of the observed conditioning variables, again via closed‑form scaling equations. As a result, each iteration is computationally cheaper and converges more rapidly than traditional conditional EM.

Both algorithms are proven to be monotonically non‑decreasing in the (conditional) log‑likelihood and to converge to a stationary point under standard regularity conditions. The paper also shows that classic IPF and EM are special cases of the proposed methods, establishing the latter as true generalizations.

Empirical evaluation is performed on synthetic data and real‑world datasets such as text classification and image labeling tasks. Results demonstrate that (1) for DCFGs, the proposed likelihood‑maximization algorithm converges faster and achieves higher final log‑likelihood values than variational Bayes and standard EM; (2) for conditional likelihood tasks, Conditional IPF reaches comparable or better predictive accuracy with fewer iterations and lower computational cost than conditional EM; and (3) the algorithms exhibit robustness to different initializations, showing stable convergence across a range of starting points.

In summary, the work delivers a unified, efficient learning framework for a broad spectrum of discrete graphical models. By defining the DCFG class and deriving closed‑form update rules for both full and conditional likelihood objectives, the authors provide a powerful tool for practitioners dealing with complex discrete structures. Future research directions include extending the framework to continuous variables, developing distributed implementations for large‑scale data, and integrating the approach with deep neural architectures to create hybrid probabilistic‑deep models.