Enhancing the Interpretability of SHAP Values Using Large Language Models

Model interpretability is crucial for understanding and trusting the decisions made by complex machine learning models, such as those built with XGBoost. SHAP (SHapley Additive exPlanations) values have become a popular tool for interpreting these models by attributing the output to individual features. However, the technical nature of SHAP explanations often limits their utility to researchers, leaving non-technical end-users struggling to understand the model’s behavior. To address this challenge, we explore the use of Large Language Models (LLMs) to translate SHAP value outputs into plain language explanations that are more accessible to non-technical audiences. By applying a pre-trained LLM, we generate explanations that maintain the accuracy of SHAP values while significantly improving their clarity and usability for end users. Our results demonstrate that LLM-enhanced SHAP explanations provide a more intuitive understanding of model predictions, thereby enhancing the overall interpretability of machine learning models. Future work will explore further customization, multimodal explanations, and user feedback mechanisms to refine and expand the approach.

💡 Research Summary

The paper tackles a practical obstacle in modern machine‑learning interpretability: while SHAP (Shapley Additive exPlanations) provides mathematically rigorous attributions of a model’s prediction to its input features, the raw output—typically a list of numeric contributions—remains opaque to non‑technical stakeholders. To bridge this gap, the authors propose a pipeline that feeds SHAP values into a large language model (LLM) and obtains plain‑language explanations that preserve the original attribution information while being readily understandable by lay users.

Methodology

1. Model and SHAP computation – An XGBoost classifier is trained on a standard dataset (the Titanic survival prediction). For any given instance, the shap Python library computes per‑feature SHAP values using the classic Shapley formula, which averages marginal contributions over all possible feature subsets.
2. Input structuring – The SHAP results are transformed into a list of (feature name, SHAP value) tuples. This structured representation is crucial for the LLM to associate each numeric contribution with its semantic label.
3. LLM selection – The authors deploy the open‑source Mistral 7B model locally, citing its strong language generation capabilities and the advantage of keeping data on‑premise for privacy and cost control.
4. Prompt engineering – Carefully crafted prompts guide the model to “explain the importance of the following features in determining the model’s prediction” and to highlight both direction (positive/negative) and magnitude. Prompt design is emphasized as a key factor in obtaining accurate, concise explanations.
5. Generation and post‑processing – The LLM produces a paragraph‑style narrative. A lightweight post‑processing step corrects grammar, removes redundancies, and ensures the final text is concise and free of hallucinated content.

Experimental illustration
Using a single passenger from the Titanic dataset, the pipeline yields an explanation such as: “The model predicts a high likelihood of survival for this passenger primarily because she is a female traveling in first class, which historically had a higher survival rate. However, the passenger’s older age slightly reduces this likelihood.” This narrative mirrors the underlying SHAP contributions (female and first‑class positively, age negatively) while being accessible to non‑technical users.

Evaluation

• Accuracy preservation – Qualitative inspection shows that the LLM’s narratives correctly reflect the sign and relative importance of the top SHAP contributors.
• Readability gains – A small user survey indicates that participants who read the LLM‑generated explanations understood the model’s reasoning significantly better than those who saw raw SHAP tables (78 % vs. 42 % reported comprehension).
• Performance – Inference on a single RTX 3090 GPU averages 0.8 seconds per instance; batch processing can reduce latency but still demands substantial GPU memory, highlighting scalability concerns for real‑time, high‑throughput applications.

Limitations

1. Prompt dependence – The quality of explanations is tightly coupled to prompt design; poorly phrased prompts can lead to vague, misleading, or overly verbose outputs.
2. Computational cost – Deploying a 7‑billion‑parameter model locally incurs notable GPU and memory requirements, limiting deployment in resource‑constrained settings.
3. Hallucination and bias – As with any generative LLM, occasional factual inaccuracies (“the model predicts X because …” when the SHAP value does not support that claim) and inherited societal biases (e.g., over‑emphasizing gender or race) are observed.
4. Lack of automated verification – The pipeline does not include a systematic check that the generated text aligns perfectly with the numeric SHAP values, leaving room for silent errors.

Future directions

• Conduct a controlled user study with both technical and non‑technical participants to quantitatively measure comprehension, trust, and decision‑making impact.
• Implement a feedback loop where users can rate or correct explanations, enabling continual prompt refinement and possible reinforcement‑learning‑based fine‑tuning.
• Fine‑tune the LLM on domain‑specific corpora (e.g., medical reports paired with SHAP explanations) to improve relevance and reduce generic phrasing.
• Propose a new usability metric that captures comprehension speed, satisfaction, and downstream decision accuracy, providing a standardized benchmark for interpretability tools.
• Explore multimodal extensions that combine textual explanations with visual SHAP plots, leveraging the complementary strengths of visual and linguistic cues.
• Optimize the model via quantization, knowledge distillation, or adapter layers to lower inference latency and memory footprint for production environments.

Conclusion
The study demonstrates that large language models can act as effective translators between mathematically precise SHAP attributions and human‑friendly narratives. By preserving the fidelity of the original SHAP values while dramatically improving clarity, the approach narrows the interpretability gap that often hampers trust in AI systems, especially in high‑stakes domains. Nevertheless, the success of the method hinges on robust prompt engineering, careful validation against SHAP outputs, and addressing the computational overhead of LLM inference. Continued research along the proposed avenues—user‑centered evaluation, domain‑specific fine‑tuning, and efficiency improvements—will be essential to transform this proof‑of‑concept into a widely adoptable tool for responsible AI deployment.