Measuring Uncertainty in Transformer Circuits with Effective Information Consistency

Mechanistic interpretability has identified functional subgraphs within large language models (LLMs), known as Transformer Circuits (TCs), that appear to implement specific algorithms. Yet we lack a formal, single-pass way to quantify when an active circuit is behaving coherently and thus likely trustworthy. Building on prior systems-theoretic proposals, we specialize a sheaf/cohomology and causal emergence perspective to TCs and introduce the Effective-Information Consistency Score (EICS). EICS combines (i) a normalized sheaf inconsistency computed from local Jacobians and activations, with (ii) a Gaussian EI proxy for circuit-level causal emergence derived from the same forward state. The construction is white-box, single-pass, and makes units explicit so that the score is dimensionless. We further provide practical guidance on score interpretation, computational overhead (with fast and exact modes), and a toy sanity-check analysis. Empirical validation on LLM tasks is deferred.

💡 Research Summary

The paper tackles the problem of quantifying the reliability of functional sub‑graphs—called Transformer Circuits (TCs)—that have been identified inside large language models (LLMs) through mechanistic interpretability. Existing work can locate a circuit that appears to implement a particular algorithm, but there is no single‑pass, mathematically grounded metric that tells whether the circuit is operating coherently on a given input. To fill this gap, the authors introduce the Effective‑Information Consistency Score (EICS), a dimension‑less scalar that can be computed from one forward pass and a small set of Jacobian‑vector products.

The construction rests on two theoretical pillars: (1) a sheaf‑theoretic notion of internal consistency, and (2) a causal‑emergence measure based on Gaussian effective information (EI). For a chosen circuit subgraph (G_M=(V_M,E_M)) the method proceeds as follows.

1. Sheaf inconsistency (C_{sh}).
   • Treat the undirected version of (G_M) as a cellular sheaf. Each node (v) carries its activation vector (a_v\in\mathbb{R}^{d_v}). Each directed edge (e=(u\to v)) carries the Jacobian (\rho_{u\to v}= \partial f_v/\partial a_u) evaluated at the current forward state.
   • The sheaf coboundary applied to the observed activations yields residuals (\rho_{u\to v}a_u - a_v). The normalized energy
     \