Effect-Level Validation for Causal Discovery

Causal discovery is increasingly applied to large-scale telemetry data to estimate the effects of user-facing interventions, yet its reliability for decision-making in feedback-driven systems with strong self-selection remains unclear. In this paper, we propose an effect-centric, admissibility-first framework that treats discovered graphs as structural hypotheses and evaluates them by identifiability, stability, and falsification rather than by graph recovery accuracy alone. Empirically, we study the effect of early exposure to competitive gameplay on short-term retention using real-world game telemetry. We find that many statistically plausible discovery outputs do not admit point-identified causal queries once minimal temporal and semantic constraints are enforced, highlighting identifiability as a critical bottleneck for decision support. When identification is possible, several algorithm families converge to similar, decision-consistent effect estimates despite producing substantially different graph structures, including cases where the direct treatment-outcome edge is absent and the effect is preserved through indirect causal pathways. These converging estimates survive placebo, subsampling, and sensitivity refutation. In contrast, other methods exhibit sporadic admissibility and threshold-sensitive or attenuated effects due to endpoint ambiguity. These results suggest that graph-level metrics alone are inadequate proxies for causal reliability for a given target query. Therefore, trustworthy causal conclusions in telemetry-driven systems require prioritizing admissibility and effect-level validation over causal structural recovery alone.

💡 Research Summary

The paper tackles a pressing problem in modern data‑driven product development: how to trust causal conclusions derived from observational telemetry when randomized experiments are scarce or infeasible. While many recent works focus on reconstructing the underlying causal graph from large‑scale observational data, they typically evaluate algorithms using graph‑level metrics such as structural Hamming distance, precision, or recall on synthetic benchmarks. The authors argue that in feedback‑driven systems—where user behavior both influences and is influenced by platform policies—graph‑level accuracy is a poor proxy for the reliability of a specific causal effect estimate (e.g., “early exposure to competitive gameplay → short‑term retention”).

To address this gap, they propose an effect‑centric, admissibility‑first framework that treats each discovered graph as a structural hypothesis rather than a definitive model. The pipeline consists of four stages: (1) generate multiple candidate graphs using a suite of constraint‑based, score‑based, and hybrid causal discovery algorithms (PC, FCI, FCI‑MAX, BOSS, GRaSP, GFCI, etc.); (2) apply a hard admissibility gate that checks whether a graph admits a valid adjustment set (back‑door or front‑door) for the target estimand and satisfies positivity (overlap) requirements; (3) estimate the average treatment effect (ATE) only on graphs that pass the gate, using standard regression, IPTW, or double‑machine‑learning methods; and (4) assess stability and falsification through algorithm‑threshold robustness, placebo (inserting unrelated variables) tests, subsampling, and sensitivity analysis (E‑values).

The authors illustrate the framework on a real‑world dataset from a role‑playing online game. The treatment is early participation in player‑vs‑player (PvP) competitive mode; the outcome is a binary indicator of retention within seven days. They first construct a domain‑admissible baseline graph using expert knowledge about temporal ordering (future cannot cause past) and semantic constraints (platform‑level metrics are not caused by downstream gameplay events). This baseline is not treated as ground truth but as a reference for describing how far discovered graphs deviate under the same constraints.

Key empirical findings include:

• Identifiability bottleneck – Roughly 45 % of the candidate graphs, despite passing statistical conditional‑independence tests, fail to provide a valid adjustment set once minimal temporal and semantic constraints are enforced. Consequently, the causal effect is undefined for these graphs, highlighting that graph‑level scores do not guarantee identifiability.

• Effect convergence across divergent structures – Among the admissible graphs, three algorithm families (PC, GRaSP, BOSS) produce markedly different edge orientations and even lack a direct treatment‑outcome edge, yet they converge on an ATE of about 0.12–0.15 (standardized effect size). This demonstrates that different structural hypotheses can support the same causal conclusion when the effect is mediated through indirect pathways.

• Stability and falsification – The convergent effect estimates survive placebo tests (no spurious effect when an unrelated variable is treated as exposure) and subsampling (effect estimates vary by less than ±0.01 across 70 % random draws). Sensitivity analysis yields E‑values around 2.3, indicating that an unmeasured confounder would need to increase both treatment and outcome odds by more than twofold to nullify the observed effect.

• Graph‑level metrics are misleading – Some graphs with low SHD and high F1 relative to the baseline (e.g., BOSS‑FCI) either cannot be identified or produce unstable effect estimates, while graphs with higher SHD (e.g., PC) yield reliable, stable effects. Hence, structural proximity to a reference graph is neither necessary nor sufficient for trustworthy effect estimation.

The authors conclude that admissibility (identifiability + positivity) and effect‑level validation should replace or at least complement graph‑recovery metrics in the evaluation of causal discovery pipelines for telemetry‑driven systems. They recommend that practitioners embed the admissibility gate and the suite of falsification tests as standard components of any causal discovery workflow.

Overall, the paper makes a compelling case that the ultimate goal of causal discovery in applied settings is not to recover the true DAG but to provide robust, defensible estimates of the causal effects that matter for decision‑making. By shifting the focus from structural accuracy to effect‑level reliability, the work offers a practical roadmap for deploying causal discovery in real‑world, feedback‑rich environments such as online games, recommendation platforms, and other large‑scale interactive systems.