Decoding Ambiguous Emotions with Test-Time Scaling in Audio-Language Models

Emotion recognition from human speech is a critical enabler for socially aware conversational AI. However, while most prior work frames emotion recognition as a categorical classification problem, real-world affective states are often ambiguous, overlapping, and context-dependent, posing significant challenges for both annotation and automatic modeling. Recent large-scale audio language models (ALMs) offer new opportunities for nuanced affective reasoning without explicit emotion supervision, but their capacity to handle ambiguous emotions remains underexplored. At the same time, advances in inference-time techniques such as test-time scaling (TTS) have shown promise for improving generalization and adaptability in hard NLP tasks, but their relevance to affective computing is still largely unknown. In this work, we introduce the first benchmark for ambiguous emotion recognition in speech with ALMs under test-time scaling. Our evaluation systematically compares eight state-of-the-art ALMs and five TTS strategies across three prominent speech emotion datasets. We further provide an in-depth analysis of the interaction between model capacity, TTS, and affective ambiguity, offering new insights into the computational and representational challenges of ambiguous emotion understanding. Our benchmark establishes a foundation for developing more robust, context-aware, and emotionally intelligent speech-based AI systems, and highlights key future directions for bridging the gap between model assumptions and the complexity of real-world human emotion.

💡 Research Summary

This paper tackles the long‑standing challenge of recognizing ambiguous, overlapping, and context‑dependent emotions in human speech. While most prior work treats speech emotion recognition (SER) as a categorical classification problem, real‑world affective states often defy clean boundaries, leading to annotation disagreement and unreliable model outputs. The authors propose a novel benchmark that combines large‑scale audio‑language models (ALMs) with test‑time scaling (TTS) techniques to improve inference‑only performance on ambiguous emotion tasks.

Problem formulation. For each utterance xₜ, M human raters provide discrete emotion labels y⁽ᵐ⁾ₜ ∈ C (K categories). The authors convert these labels into a soft probability distribution pₜ via a SoftLabel function, treating the entropy of pₜ as a quantitative measure of emotional ambiguity. The dataset D = {(xₜ, pₜ)} thus contains both the audio signal and a ground‑truth distribution over emotions. The goal is to learn a model fθ that maps audio to a predicted distribution ŷₜ ∈ Δᴷ⁻¹ and to minimize Jensen‑Shannon (JS) divergence ℓ(pₜ, ŷₜ).

Models evaluated. Eight state‑of‑the‑art ALMs are benchmarked, covering open‑source (Audio‑Flamingo 2, Qwen2.5‑Omni, Qwen2‑Audio‑Instruct, Ultravox‑Series) and closed‑source (Gemini 2.5‑pro, Gemini 2.0‑flash, GPT‑4o) systems. All models receive the same prompt template (details in the appendix) and are evaluated without any fine‑tuning on the downstream task.

Test‑time scaling strategies. Five TTS methods are systematically compared:

1. Chain‑of‑Thought (CoT) prompting – guides the model through step‑by‑step reasoning about acoustic cues before emitting a distribution.
2. Best‑of‑N (BoN) – uses beam search (beam size B = 5) to generate N candidate distributions and selects the one with the highest cumulative log‑likelihood.
3. Weighted‑BoN (W‑BoN) – treats each candidate as the mean of a Dirichlet distribution, weights them by normalized log‑likelihood, and aggregates via a Dirichlet mixture model, thereby incorporating uncertainty.
4. ALM‑verifier (ALM‑v) – a stronger ALM (GPT‑4o) acts as a scorer, evaluating each beam on criteria such as emotional consistency and audio quality, then picks the top‑scoring candidate.
5. Weighted‑ALM‑verifier (W‑ALM‑v) – similar to ALM‑v but uses the verifier’s scores as weights for a mixture aggregation.

Datasets. The benchmark spans three widely used SER corpora: IEMOCAP (4,373 utterances, 4 emotions), MSP‑Podcast (12,955 utterances, 8 emotions), and CREMA‑D (7,400 clips, 6 emotions). Each dataset exhibits varying degrees of annotator disagreement, providing a natural testbed for ambiguity analysis.

Evaluation metrics. Primary metrics are JS divergence (lower is better), Bhattacharyya Coefficient (higher is better), and R² (higher is better). In addition, the dominant predicted emotion is compared against the majority vote to compute accuracy and F1‑score, ensuring that distributional improvements also translate to correct categorical decisions.

Key findings.

• Baseline performance: Without TTS, ALMs achieve average JS scores of 0.28–0.34, with performance degrading sharply on high‑entropy (high ambiguity) samples.
• CoT prompting: Provides modest gains (~5 % JS reduction) on low‑ambiguity data but offers limited benefit when ambiguity is high.
• BoN and W‑BoN: Both improve robustness to ambiguity; W‑BoN’s Dirichlet‑based weighting yields the largest JS reduction (0.07–0.09 absolute) and consistent boosts in BC and R². The mixture approach explicitly models uncertainty, allowing the system to express multi‑emotion hypotheses.
• Verifier‑based methods: ALM‑v and W‑ALM‑v outperform pure beam‑search selection by 3–4 % lower JS, especially when the verifier is a model pre‑trained on emotion‑rich multimodal data (e.g., Gemini 2.5‑pro). Weighted verifier aggregation further refines predictions.
• Model scale interaction: Larger models (Qwen2.5‑Omni, GPT‑4o) have stronger zero‑shot capabilities, yet smaller models (Audio‑Flamingo 2) close the gap when paired with effective TTS (particularly W‑BoN). This suggests that inference‑time diversification can compensate for limited model capacity.

Implications and future work. The study demonstrates that test‑time scaling can substantially mitigate the challenges posed by ambiguous emotional cues without any additional supervised data. By generating multiple plausible outputs and aggregating them in a probabilistically sound manner, ALMs become capable of representing the inherent uncertainty of human affect. The authors propose several extensions: (1) training emotion‑specific verifier models to replace generic LLM scorers, (2) learning the Dirichlet concentration scaling τ dynamically rather than fixing it at 1, (3) integrating TTS into real‑time conversational agents where rapid adaptation to user‑specific ambiguity is critical.

Overall, this work establishes the first comprehensive benchmark for ambiguous emotion recognition with audio‑language models, systematically evaluates a suite of test‑time scaling techniques, and provides actionable insights into how model capacity, inference strategies, and the nature of emotional ambiguity interact. It paves the way for more nuanced, context‑aware, and empathetic speech‑based AI systems.