PedagoSense: A Pedology Grounded LLM System for Pedagogical Strategy Detection and Contextual Response Generation in Learning Dialogues

This paper addresses the challenge of improving interaction quality in dialogue based learning by detecting and recommending effective pedagogical strategies in tutor student conversations. We introduce PedagoSense, a pedology grounded system that combines a two stage strategy classifier with large language model generation. The system first detects whether a pedagogical strategy is present using a binary classifier, then performs fine grained classification to identify the specific strategy. In parallel, it recommends an appropriate strategy from the dialogue context and uses an LLM to generate a response aligned with that strategy. We evaluate on human annotated tutor student dialogues, augmented with additional non pedagogical conversations for the binary task. Results show high performance for pedagogical strategy detection and consistent gains when using data augmentation, while analysis highlights where fine grained classes remain challenging. Overall, PedagoSense bridges pedagogical theory and practical LLM based response generation for more adaptive educational technologies.

💡 Research Summary

PedagoSense is a novel system that bridges pedagogical theory with large language model (LLM) generation to improve the quality of tutor‑student dialogues in conversational learning environments. The architecture consists of four tightly coupled components: (1) a binary classifier that decides whether a tutor’s utterance contains any pedagogical strategy, (2) a fine‑grained multi‑class classifier that identifies the exact strategy among eight predefined categories (e.g., ask_question, explain_concept, provide_hint), (3) a strategy recommendation engine that predicts the most suitable strategy given the dialogue history, and (4) an LLM (GPT‑4o) that generates a tutor response aligned with the recommended strategy.

Data were sourced from two expert‑annotated tutor‑student corpora provided by the authors and from the Hugging Face DailyDialog dataset, which supplies non‑pedagogical conversational examples for the “no‑strategy” class. To address severe class imbalance (303 no‑strategy vs. 279 strategy examples), the authors applied two complementary balancing techniques: SMOTE, which synthetically interpolates minority samples, and GPT‑4o‑driven text augmentation, which creates high‑quality pedagogical utterances. After augmentation, each binary class contained 1,000 instances, yielding a balanced training set of 2,000 samples while preserving an untouched test set of 223 examples.

For the binary task, a BERT‑Base model was fine‑tuned (learning rate 5e‑5, batch size 16, max token length 128, early stopping). This model achieved a validation F1 of 98.85% and a test F1 of 98.5%, substantially outperforming a TF‑IDF + Logistic Regression baseline (≈95% F1). The authors also evaluated several traditional classifiers (Naïve Bayes, SVM, Random Forest, etc.) under both SMOTE and augmentation regimes, finding that data augmentation generally boosted performance for neural and linear models while degrading tree‑based models due to sensitivity to synthetic noise.

The fine‑grained classification stage employed BERT‑Large uncased, fine‑tuned on the same tokenization pipeline but with a SoftMax output over eight strategy labels. Because many strategy classes were under‑represented, the authors again used GPT‑4o to generate additional labeled examples, creating a balanced “augmented” dataset. Despite this effort, the macro F1 remained modest at 45.95% (validation) and 45.95% (test), with a clear disparity across classes: “ask_question” achieved 88.17% accuracy, whereas “provide_example” and “provide_hint” lingered around 25‑30%. This indicates that certain strategies are linguistically subtle and require richer contextual cues or auxiliary features (e.g., student performance metrics) for reliable discrimination.

Strategy recommendation was handled by a hybrid‑voting ensemble comprising an SVM, a Naïve Bayes classifier, and a Boosting model, each trained on TF‑IDF representations of the conversation history. Majority voting combined their strengths, yielding higher recommendation accuracy than any single model.

The final component integrates the recommended strategy into a prompt for GPT‑4o via an API call. The generated response is then passed through the binary BERT‑Base classifier to verify that a pedagogical strategy is indeed present. If the detected strategy matches the recommendation, the response is accepted; otherwise, it is discarded or regenerated. This double‑check mitigates the risk of LLM hallucinations or misaligned strategy usage.

Interpretability analysis using LIME on five misclassified binary examples revealed that the model overly relied on surface keywords such as “great,” leading to false positives, while words like “say” and “got” contributed negatively. This suggests that the classifier may not be capturing deeper semantic structures, a limitation that could affect downstream strategy selection.

In summary, PedagoSense demonstrates that a pipeline combining data‑balanced binary detection, fine‑grained strategy classification, ensemble recommendation, and LLM‑driven response generation can produce pedagogically informed tutor utterances. The system achieves near‑perfect performance on the binary detection task and modest gains on the multi‑class task, highlighting both the promise and the challenges of scaling such approaches. Future work should focus on expanding annotated corpora, incorporating multimodal signals (e.g., student affect, problem difficulty), and exploring continual learning to adapt to evolving tutoring contexts.