Evaluating an evidence-guided reinforcement learning framework in aligning light-parameter large language models with decision-making cognition in psychiatric clinical reasoning

Large language models (LLMs) hold transformative potential for medical decision support yet their application in psychiatry remains constrained by hallucinations and superficial reasoning. This limitation is particularly acute in light-parameter LLMs which are essential for privacy-preserving and efficient clinical deployment. Existing training paradigms prioritize linguistic fluency over structured clinical logic and result in a fundamental misalignment with professional diagnostic cognition. Here we introduce ClinMPO, a reinforcement learning framework designed to align the internal reasoning of LLMs with professional psychiatric practice. The framework employs a specialized reward model trained independently on a dataset derived from 4,474 psychiatry journal articles and structured according to evidence-based medicine principles. We evaluated ClinMPO on a unseen subset of the benchmark designed to isolate reasoning capabilities from rote memorization. This test set comprises items where leading large-parameter LLMs consistently fail. We compared the ClinMPO-aligned light LLM performance against a cohort of 300 medical students. The ClinMPO-tuned Qwen3-8B model achieved a diagnostic accuracy of 31.4% and surpassed the human benchmark of 30.8% on these complex cases. These results demonstrate that medical evidence-guided optimization enables light-parameter LLMs to master complex reasoning tasks. Our findings suggest that explicit cognitive alignment offers a scalable pathway to reliable and safe psychiatric decision support.

💡 Research Summary

This paper tackles the persistent problem that lightweight large language models (LLMs) – essential for privacy‑preserving, low‑resource clinical deployment – still suffer from hallucinations and shallow, memorization‑driven reasoning when applied to psychiatric decision‑making. The authors introduce ClinMPO, a reinforcement‑learning (RL) framework that aligns a light‑parameter LLM’s internal reasoning with the cognitive processes used by professional psychiatrists.

ClinMPO consists of three tightly coupled components. First, an “Evidence Dataset” was built from 4,474 peer‑reviewed psychiatry journal articles, yielding 18,569 entries that are organized according to the Oxford Centre for Evidence‑Based Medicine hierarchy. Second, a dedicated reward model, ClinRM, is trained independently on this dataset. ClinRM scores each generated reasoning trajectory not for linguistic fluency but for clinical soundness: it rewards correct symptom integration, longitudinal context handling, and appropriate citation of high‑level evidence, while penalizing symptom misidentification, faulty pathogenesis, incorrect differential diagnoses, and unsuitable treatment plans. Third, a multi‑group policy optimization (GRPO) algorithm uses relative advantages among candidate responses within each case group, rather than absolute reward values, to update the policy. This design discourages shortcuts such as verbosity or stylistic imitation and pushes the model toward reasoning patterns that mirror expert psychiatric judgment.

Evaluation was performed on a “hard” subset of 8,849 questions drawn from publicly available medical datasets. The subset was deliberately chosen because leading large‑parameter LLMs consistently fail on these items, thereby isolating genuine reasoning ability from memorization. The same test set was administered to 300 medical students, establishing a human benchmark (average accuracy = 30.8%). Four scales of the open‑source Qwen‑3 model (0.6 B, 1.7 B, 4 B, 8 B) were fine‑tuned under four strategies: base (no further training), supervised fine‑tuning (SFT), conventional group‑policy RL (GRPO), and the proposed ClinMPO.

The 8 B Qwen‑3 model trained with ClinMPO achieved an overall diagnostic accuracy of 31.43%, surpassing the human reference and outperforming all non‑ClinMPO baselines (GRPO = 30.80%, SFT = 28.27%, base = 28.27%). Accuracy gains grew with model size: +0.23 % at 0.6 B, +1.84 % at 1.7 B, +5.64 % at 4 B, and +3.17 % at 8 B, yielding an average improvement of 2.72 percentage points across scales—larger than the improvements from SFT (1.64 pp) or GRPO (1.80 pp).

Fine‑grained analysis across 26 ICD‑11 diagnostic categories and 12 psychiatric practice competencies showed that ClinMPO consistently leads or matches the best baseline. Notably, in “Mental, behavioural or neurodevelopmental disorders” and “Impulse control disorders” the 4 B model reached 50 % accuracy, far above the 27.27 % and 20.90 % achieved by students. In competency areas such as “Monitoring, Follow‑up & Measurement‑Based Care” and “Comorbidity & Complexity Management,” the 8 B ClinMPO model attained 42.86 % and 44.74 % respectively, again exceeding human performance.

Error‑transition analysis (false‑to‑true vs. true‑to‑false) demonstrated that ClinMPO corrects a substantial proportion of previously incorrect predictions (e.g., 22.6 % correction at 4 B, compared with 21.4 % for GRPO and 13.9 % for SFT), confirming genuine reasoning improvement rather than random chance. Distributional plots revealed higher medians, tighter inter‑quartile ranges, and fewer extreme low‑performing tails for ClinMPO across all scales, indicating superior robustness and consistent generalization across heterogeneous clinical tasks.

The authors acknowledge limitations: the Evidence Dataset is psychiatry‑specific, so transfer to other specialties remains untested; the reward model relies on expert annotations, which may embed bias; and a 31 % accuracy level is still insufficient for autonomous clinical use, necessitating human oversight and additional safety layers.

In summary, ClinMPO provides a novel, evidence‑guided RL paradigm that successfully aligns lightweight LLMs with professional psychiatric cognition. By rewarding evidence‑based logical chains and employing relative policy updates, the framework enables models with as few as 0.6 B parameters to approach, and in some domains exceed, human diagnostic performance. This work charts a scalable path toward safe, efficient AI‑augmented mental‑health decision support, while highlighting the need for broader validation, bias mitigation, and rigorous clinical safety mechanisms before real‑world deployment.