Exploring the Global-to-Local Attention Scheme in Graph Transformers: An Empirical Study

Graph Transformers (GTs) show considerable potential in graph representation learning. The architecture of GTs typically integrates Graph Neural Networks (GNNs) with global attention mechanisms either in parallel or as a precursor to attention mechanisms, yielding a local-and-global or local-to-global attention scheme. However, as the global attention mechanism primarily captures long-range dependencies between nodes, these integration schemes may suffer from information loss, where the local neighborhood information learned by GNN could be diluted by the attention mechanism. Therefore, we propose G2LFormer, featuring a novel global-to-local attention scheme where the shallow network layers use attention mechanisms to capture global information, while the deeper layers employ GNN modules to learn local structural information, thereby preventing nodes from ignoring their immediate neighbors. An effective cross-layer information fusion strategy is introduced to allow local layers to retain beneficial information from global layers and alleviate information loss, with acceptable trade-offs in scalability. To validate the feasibility of the global-to-local attention scheme, we compare G2LFormer with state-of-the-art linear GTs and GNNs on node-level and graph-level tasks. The results indicate that G2LFormer exhibits excellent performance while keeping linear complexity.

💡 Research Summary

Graph Transformers (GTs) have emerged as a powerful alternative to traditional Graph Neural Networks (GNNs) because their global self‑attention can directly model long‑range dependencies, alleviating the over‑smoothing and over‑squashing problems that plague message‑passing GNNs. However, most existing GT designs place the global attention modules toward the deeper layers of the network, either in parallel with local GNN layers (the “local‑and‑global” scheme) or preceding them (the “local‑to‑global” scheme). Recent analyses reveal that this ordering leads to an “over‑globalizing” effect: the final node representations become dominated by distant interactions, while useful local structural cues are under‑exploited.

The paper introduces a fundamentally different architecture called global‑to‑local. In this scheme, a shallow global attention block captures a coarse, whole‑graph context first, and deeper layers consist of conventional GNN modules that refine the representation by aggregating information from immediate neighbors. The authors instantiate this idea in a model named G2LFormer. The global block uses the linear‑attention mechanism from SGFormer, which avoids the quadratic cost of classic Transformers and runs in O(N) time and memory. Only a single global layer is employed, guaranteeing that the global context is injected efficiently.

A central challenge of the global‑to‑local design is preserving the information learned by the global layer when it is passed to the subsequent GNN layers. To address this, the authors adopt a cross‑layer information‑fusion strategy called NOSAF (Node‑Specific Layer Aggregation and Filtration). NOSAF concatenates the global representation with the current GNN hidden state, computes a node‑wise importance score γ via a two‑layer MLP with LeakyReLU and sigmoid activations, and then filters the GNN input by element‑wise multiplication with γ. This dynamic weighting re‑allocates node contributions, mitigating both over‑smoothing in the GNN and over‑globalization in the attention block, without adding extra parameters that would increase asymptotic complexity.

The overall pipeline can be summarized as:

1. Global Layer – compute Q, K, V from raw node features, normalize them with Frobenius norm, derive a diagonal scaling matrix D, and obtain the global embedding h_TL via a single‑head linear attention followed by a feed‑forward network.
2. NOSAF Fusion – combine h_TL with each subsequent GNN hidden state, generate γ, and filter the hidden state before it enters the next GNN layer.
3. Local Layers – apply either Cluster‑GCN (which partitions the graph to improve scalability) or GatedGCN (which uses gating to control message flow) for n deep layers, producing the final node embeddings h_GL.

The authors provide a theoretical analysis showing that the global attention’s computational cost drops from O(N²) to O(N) thanks to the linear‑attention formulation, and that the GNN part retains its linear or near‑linear scaling.

Empirical evaluation covers both node‑level tasks (Cora, Citeseer, PubMed) and graph‑level tasks (ZINC, ogbg‑molhiv, and several benchmark datasets). Baselines include state‑of‑the‑art linear GTs (SGFormer, Polynormer), classic GNNs (Cluster‑GCN, GatedGCN), and hybrid GT‑GNN models that follow the local‑and‑global or local‑to‑global schemes (e.g., GraphGPS, GraphTransformer). Across the board, G2LFormer achieves higher accuracy or lower error than all baselines, typically improving by 1–3 percentage points. Notably, on larger graphs the model’s memory footprint and inference time are substantially lower than quadratic GTs, confirming the claimed linear scalability. Ablation studies demonstrate that removing the NOSAF fusion degrades performance, highlighting its importance for preserving global information.

While the results are compelling, the paper has some limitations. First, the global component is limited to a single linear‑attention layer; more complex global patterns (e.g., multi‑community structures) might require deeper or multi‑head attention. Second, the NOSAF module introduces additional matrix multiplications and MLPs whose exact parameter count and runtime overhead are not fully quantified, leaving open questions about efficiency on truly massive graphs (millions of nodes). Third, the experimental suite, though diverse, does not include ultra‑large industrial graphs where scalability claims would be most critical.

In conclusion, the work proposes a novel architectural paradigm—global‑to‑local attention—that reverses the conventional ordering of global and local modules in graph Transformers. By coupling a lightweight linear global attention with deep GNN layers and a dynamic cross‑layer fusion mechanism, G2LFormer delivers state‑of‑the‑art performance on a range of benchmarks while preserving linear computational complexity. The paper opens a new direction for designing scalable, expressive graph models and suggests several avenues for future research, such as deeper global stacks, more efficient fusion mechanisms, and validation on billion‑edge graphs.