Higher-Order Modular Attention: Fusing Pairwise and Triadic Interactions for Protein Sequences

Transformer self-attention computes pairwise token interactions, yet protein sequence to phenotype relationships often involve cooperative dependencies among three or more residues that dot product attention does not capture explicitly. We introduce Higher-Order Modular Attention, HOMA, a unified attention operator that fuses pairwise attention with an explicit triadic interaction pathway. To make triadic attention practical on long sequences, HOMA employs block-structured, windowed triadic attention. We evaluate on three TAPE benchmarks for Secondary Structure, Fluorescence, and Stability. Our attention mechanism yields consistent improvements across all tasks compared with standard self-attention and efficient variants including block-wise attention and Linformer. These results suggest that explicit triadic terms provide complementary representational capacity for protein sequence prediction at controllable additional computational cost.

💡 Research Summary

The paper introduces Higher‑Order Modular Attention (HOMA), a novel attention operator designed to capture both pairwise (2‑D) and explicit triadic (3‑D) interactions in protein sequences. Traditional transformer self‑attention computes only pairwise dot‑product affinities, which limits its ability to model epistatic effects where the impact of a mutation depends on multiple residues simultaneously. HOMA augments the standard attention pathway with a second pathway that computes a triadic score S³_{ijk}= (1/√d) Σ_c Q_{ic} K_{jc} U_{kc}. After softmax normalization over ordered pairs (j,k), the model aggregates element‑wise products of value vectors V_j⊙V_k, yielding a triadic output O³_i. The pairwise and triadic outputs are concatenated and fused through a small MLP, producing the final token representation.

Because naïve triadic computation scales cubically (O(L³)), the authors propose three efficiency tricks. First, the sequence is partitioned into overlapping blocks of length ℓ with stride s; attention is computed independently within each block. Second, within each block a local window of size w (w≪ℓ) restricts the (j,k) pairs considered for each query, reducing the per‑block triadic cost to O(ℓ·w²). Third, the projection matrix for the U pathway is factorized into a low‑rank form (rank r≪d), limiting additional parameters. Overall, the pairwise component costs O(T·ℓ²) and the triadic component O(T·ℓ·w²), where T is the number of blocks, making the method feasible for proteins up to several hundred residues.

Experiments are conducted on three TAPE benchmarks: secondary‑structure classification (per‑residue Q3 accuracy), fluorescence regression, and stability regression (both evaluated with Spearman ρ). All models share the same 12‑layer transformer backbone (d_model=512, 8 heads, FFN dim=1024, dropout=0.4) and differ only in the attention module. Baselines include global multi‑head self‑attention (Pairwise‑2D), overlapping block‑wise attention (Blockwise‑2D), and Linformer‑style low‑rank attention (Linear‑2D). HOMA builds on the Blockwise‑2D backbone and adds the triadic pathway; window sizes w∈{3,5,7} are explored.

Results show consistent improvements across tasks. For secondary structure, Blockwise‑2D already outperforms the global baseline (e.g., CASP12 accuracy 0.6368 vs. 0.5582). HOMA further raises accuracy to 0.6588 with w=5 (≈3.5% relative gain). Similar trends appear on CB513 and TS115, where the best HOMA configurations achieve accuracies of 0.6504 and 0.6789 respectively. In regression, HOMA improves stability correlation from 0.6509 (Blockwise‑2D) to 0.7152 (≈9.9% relative gain) with w=5, and fluorescence correlation from 0.6998 to 0.6821 (≈5% gain) with w=7. Computationally, HOMA adds only ~2% more parameters and maintains token‑processing throughput comparable to Blockwise‑2D; smaller windows further reduce memory and increase speed.

The authors acknowledge limitations: the triadic pathway is confined to local windows, so long‑range three‑way dependencies are not directly modeled, and performance is sensitive to block and window hyperparameters. Future work could incorporate dynamic windows, multi‑scale block hierarchies, or hypergraph‑based global triadic connections, as well as extending HOMA to other biological sequences (DNA, RNA) or large‑scale language models. Overall, the study demonstrates that explicit higher‑order attention can enrich protein representations while keeping computational costs tractable.