HetGCoT: Heterogeneous Graph-Enhanced Chain-of-Thought LLM Reasoning for Academic Question Answering

Academic question answering (QA) in heterogeneous scholarly networks presents unique challenges requiring both structural understanding and interpretable reasoning. While graph neural networks (GNNs) capture structured graph information and large language models (LLMs) demonstrate strong capabilities in semantic comprehension, current approaches lack integration at the reasoning level. We propose HetGCoT, a framework enabling LLMs to effectively leverage and learn information from graphs to reason interpretable academic QA results. Our framework introduces three technical contributions: (1) a framework that transforms heterogeneous graph structural information into LLM-processable reasoning chains, (2) an adaptive metapath selection mechanism identifying relevant subgraphs for specific queries, and (3) a multi-step reasoning strategy systematically incorporating graph contexts into the reasoning process. Experiments on OpenAlex and DBLP datasets show our approach outperforms all sota baselines. The framework demonstrates adaptability across different LLM architectures and applicability to various scholarly question answering tasks.

💡 Research Summary

HetGCoT (Heterogeneous Graph‑Enhanced Chain‑of‑Thought) is a novel framework that tightly integrates heterogeneous academic graphs with large language models (LLMs) to tackle scholarly question answering (QA) tasks. The authors identify three core challenges in academic QA: (1) modeling the multi‑type entities and relations inherent in scholarly networks, (2) dynamically selecting task‑relevant subgraphs rather than processing the entire graph, and (3) converting structural knowledge into natural‑language explanations that can be understood and trusted by users.

Graph Construction and Embedding
The system builds a heterogeneous graph containing papers (P), authors (A), and venues (V) with two relation types: paper‑venue and paper‑author. Node features combine textual embeddings from Sentence‑BERT (titles, abstracts, keywords) with numerical attributes (citation counts, impact factors) via concatenation and LayerNorm. These initial embeddings are fed into a Heterogeneous Graph Transformer (HGT), which applies type‑aware attention to produce rich, context‑sensitive node representations.

Adaptive Metapath Selection
To extract meaningful substructures, the authors define four metapath templates—APVP_A, VPAPV, AP_A, and OAPVP_AO—that capture venue‑based author connections, shared‑venue links, direct collaborations, and institutional pathways. For a given query node, candidate metapath instances are generated by locating semantically similar nodes using cosine similarity on HGT embeddings. FastGTN, a graph‑convolutional architecture, is then trained in an unsupervised reconstruction setting to learn relation‑importance weights. These learned weights are used to score each metapath instance (with optional length normalization), and a stratified top‑k (k = 5) selection is performed per template to guarantee diversity.

Metapath Naturalization and Chain‑of‑Thought Prompting
Selected metapaths are transformed into natural‑language sentences via template‑based verbalization. Each sentence is prefixed with a confidence score derived from FastGTN, allowing the LLM to weigh evidence. The authors design a four‑step CoT reasoning pipeline: (1) Graph Structure Analysis – interpret the naturalized metapaths; (2) Content Analysis – examine the target paper’s textual and numeric attributes; (3) Collaboration/Institution Analysis – leverage author‑centric metapaths; (4) Answer Generation – synthesize insights into a final answer with an evidence‑backed explanation. The prompt includes a system message defining the model as an academic QA expert and a user message that feeds the four reasoning stages together with the metapath descriptions.

LLM Enhancement and Training
GPT‑4o mini is fine‑tuned on a curated dataset of structured reasoning examples across multiple scholarly QA tasks. The loss function incorporates the confidence‑weighted probability of the correct answer, encouraging the model to prioritize high‑confidence metapaths during generation. During inference, the same prompting schema is applied to any LLM, demonstrating the framework’s model‑agnostic nature.

Experiments
The framework is evaluated on two large scholarly corpora: a filtered OpenAlex subgraph (≈ 76 k nodes, 105 k edges) limited to high‑rank venues, and a DBLP subgraph (≈ 62 k nodes, 80 k edges). Three QA scenarios are tested: venue recommendation, author‑paper matching, and collaboration discovery. HetGCoT achieves 92.21 % and 83.70 % Hit@1 for venue recommendation on OpenAlex and DBLP respectively, outperforming state‑of‑the‑art baselines such as heterogeneous GNNs, Graph‑Prompt, Graph‑CoT, and retrieval‑augmented methods by margins ranging from 7 % to 15 % absolute. Ablation studies reveal that (i) removing the confidence weighting drops performance by ~12 %, (ii) using a single generic metapath template reduces accuracy by ~9 %, and (iii) omitting the multi‑step CoT decomposition leads to less interpretable and less accurate answers.

Insights and Limitations
HetGCoT demonstrates that converting graph structures into confidence‑weighted natural language enables LLMs to perform deep, transparent reasoning over scholarly networks. The adaptive metapath selection ensures that only the most relevant subgraphs are presented, reducing prompt length while preserving essential context. However, the current set of four metapath templates may not capture more intricate scholarly relations such as citation cascades over time or funding‑project links, suggesting future work on automated template discovery. Additionally, candidate metapath generation can become computationally expensive on very large graphs, motivating research into scalable sampling strategies.

In summary, HetGCoT bridges the gap between structural graph reasoning and large‑scale language understanding by embedding heterogeneous graph evidence directly into the chain‑of‑thought process of LLMs. This integration yields superior accuracy, richer explanations, and broad applicability across different LLM architectures, marking a significant step forward for AI‑driven academic question answering.