Causal Preference Elicitation

We propose causal preference elicitation, a Bayesian framework for expert-in-the-loop causal discovery that actively queries local edge relations to concentrate a posterior over directed acyclic graphs (DAGs). From any black-box observational posterior, we model noisy expert judgments with a three-way likelihood over edge existence and direction. Posterior inference uses a flexible particle approximation, and queries are selected by an efficient expected information gain criterion on the expert’s categorical response. Experiments on synthetic graphs, protein signaling data, and a human gene perturbation benchmark show faster posterior concentration and improved recovery of directed effects under tight query budgets.

💡 Research Summary

The paper introduces CaPE (Causal Preference Elicitation), a Bayesian framework that integrates expert feedback into causal discovery in a sequential, active‑learning fashion. Starting from any black‑box observational posterior over weighted DAGs, q₀(W|X), obtained by standard methods (MCMC, bootstrap, variational inference, etc.), CaPE models an expert’s noisy judgments as a three‑valued categorical matrix Y∈{0,1,2}^{D×D}. The values encode “i→j”, “j→i”, or “no edge”. A hierarchical logistic likelihood pθ(Yij|W) maps the latent weighted graph to these categories. For each ordered pair (i,j) a local direction score s_{i→j}(W)=g(|W_{ij}|^γ)+λϕ_{i→j}(W) is computed, where g is a monotone link (log in experiments), γ scales edge magnitude, and ϕ can incorporate structural cues such as v‑structure support or cycle risk. Two summary statistics are derived: a_{ij}=max(s_{i→j},s_{j→i}) (absolute evidence of any edge) and d_{ij}=s_{i→j}−s_{j→i} (relative evidence for orientation). Edge existence probability is p_edge=σ(β_edge·a_{ij}), and conditional on existence the orientation probability is p_{i→j|edge}=σ(β_dir·d_{ij}). The three‑class probabilities become: P(Yij=2)=1−p_edge, P(Yij=1)=p_edge·p_{i→j|edge}, P(Yij=0)=p_edge·(1−p_{i→j|edge}). Parameters β_edge and β_dir capture the expert’s reliability and decisiveness; λ and γ control the influence of edge magnitude and structural features.

Posterior updating proceeds with a particle filter. An initial set of S particles {W^{(s)}} is drawn from q₀, each with equal weight. When a new expert response Y^{(t)}{i_t j_t} is observed, particle weights are multiplied by the corresponding likelihood term pθ(Y^{(t)}{i_t j_t}|W^{(s)}). To avoid weight degeneracy, the effective sample size (ESS) is monitored; if ESS falls below a fraction δ_s of S, systematic resampling is performed. After resampling, a lightweight Metropolis–Hastings rejuvenation kernel proposes small graph edits (add/remove a single directed edge, flip orientation, or adjust an edge weight). Proposals that create cycles are rejected outright; otherwise they are accepted with the usual MH ratio that preserves the full posterior (including the product of all observed likelihood terms). This combination of importance weighting, ESS‑triggered resampling, and rejuvenation yields a flexible approximation to the evolving posterior q_t(W) while maintaining acyclicity and particle diversity.

Query selection is driven by an expected information gain (EIG) criterion derived from the BALD (Bayesian Active Learning by Disagreement) framework. For each candidate ordered pair (i,j), the algorithm computes the current posterior predictive distribution over Y_{ij} and estimates the reduction in posterior entropy that would result from observing that entry. Because Y_{ij} is a low‑dimensional categorical variable, the EIG can be evaluated efficiently, allowing the method to select the most globally informative edge query under a strict budget. This active strategy contrasts with naïve random querying or simple uncertainty‑based heuristics.

Empirical evaluation covers three settings: (1) synthetic DAGs with 20–50 nodes under varying edge densities and simulated expert error rates, (2) a real protein signaling network where ground‑truth causal edges are known from experimental biology, and (3) a human gene perturbation benchmark containing experimentally validated causal effects. Baselines include random query selection, variance‑based uncertainty queries, and static expert constraints (hard forbidden edges). Across all experiments, CaPE achieves substantially faster posterior concentration (30–45 % reduction in average entropy for the same number of queries), lower Structural Hamming Distance, and higher edge‑orientation AUC compared to baselines. Notably, the method remains robust when the expert’s error rate is as high as 10–20 %.

The key contributions are: (i) a general probabilistic framework that fuses any observational posterior with sequential expert feedback, (ii) a three‑way hierarchical logistic model that captures noisy, biased human judgments while allowing interpretable hyper‑parameters, (iii) a particle‑based Bayesian updating scheme with resampling and rejuvenation that scales to moderate‑size DAGs, and (iv) a BALD‑style EIG acquisition function that makes active learning tractable in the combinatorial space of DAGs. By enabling efficient, budget‑aware interaction with domain experts, CaPE bridges the gap between data‑driven causal inference and expert intuition, offering a practical tool for scientific domains where experiments are costly and expert knowledge is valuable.