Knowledge Synthesis Graph: An LLM-Based Approach for Modeling Student Collaborative Discourse

Asynchronous, text-based discourse-such as students’ posts in discussion forums-is widely used to support collaborative learning. However, the distributed and evolving nature of such discourse often makes it difficult to see how ideas connect, develop, and build on one another over time. As a result, learners may struggle to recognize relationships among ideas-a process that is critical for idea advancement in productive collaborative discourse. To address this challenge, we explore how large language models (LLMs) can provide representational guidance by modeling student discourse as a Knowledge Synthesis Graph (KSG). The KSG identifies ideas from student discourse and visualizes their epistemic relationships, externalizing the current state of collaborative knowledge in a form that can support further inquiry and idea advancement. In this study, we present the design of the KSG and evaluate the LLM-based approach for constructing KSGs from authentic student discourse data. Through multi-round human-expert coding and prompt iteration, our results demonstrate the feasibility of using our approach to construct reliable KSGs across different models. This work provides a technical foundation for modeling collaborative discourse with LLMs and offers pedagogical implications for augmenting complex knowledge work in collaborative learning environments.

💡 Research Summary

This paper introduces the Knowledge Synthesis Graph (KSG), a novel representation for modeling asynchronous, text‑based collaborative discourse in education, and demonstrates how large language models (LLMs) can be harnessed to construct it automatically. The KSG consists of three interconnected components: (1) Micro‑ideas, which are concise, stand‑alone reformulations of students’ raw forum or annotation posts; (2) Synthesis Nodes, which are higher‑level concepts extracted from the course readings; and (3) Epistemic Relations, which link each Micro‑idea to one or more Synthesis Nodes, indicating both the directional stance (build‑toward vs. push‑back) and the functional role (evidence, counter‑example, elaboration, deconstruction, new idea, question/critique).
The authors design a three‑stage pipeline. In Stage 1, an LLM receives the original student comment, the relevant passage from the textbook, and any reply context, then performs a “Contextualize → Filter → Rewrite” operation to produce a Micro‑idea and assign it one of four epistemic labels (descriptive, interpretive, analytical, generative). Prompt engineering is iteratively refined across three versions (P_base, P1, P2), yielding substantial gains in inter‑rater reliability (Cohen’s κ from 0.619 to 0.643) and balanced F1 scores (Macro F1 from 0.561 to 0.722).
In Stage 2, the same or a similar LLM ingests the full reading text, a concise summary, and instructor‑provided prompts to generate a set of Synthesis Nodes that capture the core arguments and concepts of the material. Experiments reveal that a hybrid prompt—summary plus targeted instructor cues—produces the most coherent and pedagogically aligned nodes, whereas using only the abstract leads to shallow nodes and feeding the entire unstructured text invites hallucinations.
Stage 3 links Micro‑ideas to Synthesis Nodes. The authors propose a two‑level coding scheme: Level 1 encodes stance (+ or –), and Level 2 specifies one of six functional categories. Four prompt configurations (P_base, P1, P2, P3) are tested across four LLM variants (GPT‑4o and three GPT‑5 series models). Metrics focus on execution rate (percentage of valid outputs) and cross‑model consistency (frequency of identical outputs among models). The proposed two‑level scheme (P3) outperforms the others, achieving the highest execution rate and the greatest consistency, indicating that a well‑structured relational schema stabilizes LLM behavior even when the task is inherently open‑ended.
The evaluation uses a real dataset of 42 social‑annotation entries from a graduate‑level instructional design course. Human expert coders provide ground truth for Micro‑idea labeling, Synthesis Node relevance, and Epistemic Relation assignment. Quantitative results are complemented by qualitative analyst memos that highlight common error patterns (e.g., over‑generalization, missed nuance, or mis‑classification of question‑type contributions).
Discussion addresses two major concerns about deploying LLMs in learning environments. First, LLMs tend to amplify dominant statistical patterns, potentially marginalizing minority perspectives and flattening productive contradictions. Second, there is a risk that AI‑generated artifacts could position learners as passive recipients rather than active knowledge co‑creators. The authors argue that careful prompt design—explicitly preserving epistemic diversity—and aligning model outputs with pedagogical intent can mitigate these risks. They also critique conventional evaluation that focuses solely on accuracy against a single ground truth, advocating for frameworks that capture interaction dynamics, contextual relevance, and learning impact.
Future work is outlined along three axes: (1) classroom‑scale deployments to observe how students interact with and benefit from KSGs in authentic settings; (2) incorporation of advanced reasoning techniques such as chain‑of‑thought prompting and verifier models to improve the reliability and interpretability of Epistemic Relation linking; and (3) reconceptualizing the KSG as a dynamic, collaboratively edited artifact that evolves alongside the discourse, turning the AI‑generated graph into a prompt for further elaboration rather than a static end product.
Overall, the study provides a proof‑of‑concept that LLMs can automatically synthesize and structure collaborative student discourse into a graph that makes ideas visible, connects them to canonical course concepts, and encodes nuanced epistemic relationships. By combining rigorous prompt engineering, multi‑stage pipeline design, and thorough human‑in‑the‑loop evaluation, the work lays a technical and pedagogical foundation for future AI‑enhanced collaborative learning tools.