RE-LLM: Refining Empathetic Speech-LLM Responses by Integrating Emotion Nuance

With generative AI advancing, empathy in human-AI interaction is essential. While prior work focuses on emotional reflection, emotional exploration, key to deeper engagement, remains overlooked. Existing LLMs rely on text which captures limited emotion nuances. To address this, we propose RE-LLM, a speech-LLM integrating dimensional emotion embeddings and auxiliary learning. Experiments show statistically significant gains in empathy metrics across three datasets. RE-LLM relatively improves the Emotional Reaction score by 14.79% and 6.76% compared to text-only and speech-LLM baselines on ESD. Notably, it raises the Exploration score by 35.42% and 3.91% on IEMOCAP, 139.28% and 9.83% on ESD, and 60.95% and 22.64% on MSP-PODCAST. It also boosts unweighted accuracy by 5.4% on IEMOCAP, 2.3% on ESD, and 6.9% on MSP-PODCAST in speech emotion recognition. These results highlight the enriched emotional understanding and improved empathetic response generation of RE-LLM.

💡 Research Summary

The paper introduces RE‑LLM, a speech‑enabled large language model (LLM) that incorporates fine‑grained emotional nuance from the acoustic signal to improve empathetic dialogue generation. While prior empathetic AI systems mainly focus on “emotional reflection” – matching the user’s expressed affect – they largely ignore “emotional exploration,” the process of asking clarifying or probing questions that deepens the interaction. The authors argue that this omission limits the depth of human‑AI empathy, especially in counseling‑style conversations where exploration is crucial.

Architecture.
RE‑LLM builds on a standard speech‑LLM pipeline (Whisper‑large‑v2 for acoustic feature extraction and Qwen‑7B‑Chat as the generative LLM) and adds an “emotion nuance module.” This module consists of a frozen wav2vec 2.0‑based emotion encoder that outputs three‑dimensional (valence, arousal, dominance) embeddings for each time step. These embeddings are concatenated with the raw Whisper embeddings, producing a richer, emotion‑aware representation. A modality adapter (three 1‑D convolution layers with a bottleneck of 512 hidden units) compresses the concatenated sequence into a format the LLM can ingest.

Training strategy.
Training proceeds in two stages. First, the authors generate “expected responses” by prompting the LLM with a template that includes an explicit emotion tag (e.g., “Continue the following sentence that reflects a emotion tone”). These expected responses serve as targets for alignment. Second, the model is trained to minimize a combined loss: (1) KL‑divergence between the generated response and the expected response, (2) cross‑entropy loss for categorical emotion classification (four classes: neutral, happy, angry, sad), and (3) mean‑squared error loss for dimensional emotion regression (valence, arousal, dominance). The auxiliary tasks are applied only during training; at inference time the model simply generates a response conditioned on the speech input.

Datasets and evaluation.
The authors evaluate on three benchmark emotion corpora: IEMOCAP (English dyadic speech, 4 categorical emotions with dimensional labels), the English portion of the Emotional Speech Dataset (ESD, 350 utterances, categorical only; pseudo‑dimensional labels are generated using the same wav2vec model), and MSP‑PODCAST (large “in‑the‑wild” podcast excerpts with both categorical and dimensional annotations). Empathetic performance is measured with the EPITOME‑based automatic scoring system, which yields two scores per response: Emotional Reaction (ER) – how well the model reacts to the user’s affect, and Exploration (Ex) – how effectively the model asks probing or clarifying questions. Both scores range from 0 (poor) to 2 (excellent). Additionally, unweighted accuracy (UA) of emotion recognition is reported.

Results.
Across all three datasets, RE‑LLM outperforms four baselines: (a) Text‑only LLM (ground‑truth transcripts), (b) Whisper‑transcribed text fed to LLM, (c) Text‑LLM with prepended categorical emotion labels, and (d) the prior speech‑LLM BLSP‑Emo (both fine‑tuned and frozen versions). Key findings include:

• Emotional Reaction (ER): On ESD, RE‑LLM improves ER by 14.79 % over the text‑only baseline and by 6.76 % over the speech‑LLM baseline. Gains on IEMOCAP and MSP‑PODCAST are smaller but still statistically significant (p < 0.05).
• Exploration (Ex): The most striking improvements appear in the exploration metric. RE‑LLM raises Ex by 35.42 % (IEMOCAP), 139.28 % (ESD), and 60.95 % (MSP‑PODCAST) relative to the text‑only model, and by 3.91 %, 9.83 %, and 22.64 % respectively compared to the speech‑LLM baseline.
• Emotion Recognition (UA): Unweighted accuracy improves by 5.4 % (IEMOCAP), 2.3 % (ESD), and 6.9 % (MSP‑PODCAST) over the BLSP‑Emo baseline, indicating that the auxiliary tasks indeed sharpen the model’s perception of affect.

Ablation studies confirm the importance of both components: removing the dimensional emotion auxiliary task degrades exploration and UA scores, while omitting the frozen emotion encoder leads to a collapse in overall performance.

Discussion.
The authors acknowledge several limitations. First, for datasets lacking dimensional annotations (e.g., ESD), pseudo‑labels generated by the wav2vec model may introduce noise; the impact of label quality on downstream performance warrants further study. Second, the expected‑response generation relies on static prompts, which may not capture the dynamic nature of multi‑turn conversations. Third, the combined Whisper + wav2vec + Qwen architecture is computationally heavy, posing challenges for real‑time deployment.

Conclusion and future work.
RE‑LLM demonstrates that integrating fine‑grained acoustic emotion cues and dual‑objective auxiliary training can substantially enhance both affective reaction and exploratory questioning in empathetic AI. This work moves beyond simple emotion mirroring toward a more nuanced, dialogically rich form of empathy. Future directions include (a) leveraging reinforcement learning to adapt exploration strategies in real time, (b) extending the framework to multi‑turn, context‑aware dialogues, and (c) exploring model compression techniques to enable on‑device deployment.