EMO-TTA: Improving Test-Time Adaptation of Audio-Language Models for Speech Emotion Recognition

Speech emotion recognition (SER) with audio-language models (ALMs) remains vulnerable to distribution shifts at test time, leading to performance degradation in out-of-domain scenarios. Test-time adaptation (TTA) provides a promising solution but often relies on gradient-based updates or prompt tuning, limiting flexibility and practicality. We propose Emo-TTA, a lightweight, training-free adaptation framework that incrementally updates class-conditional statistics via an Expectation-Maximization procedure for explicit test-time distribution estimation, using ALM predictions as priors. Emo-TTA operates on individual test samples without modifying model weights. Experiments on six out-of-domain SER benchmarks show consistent accuracy improvements over prior TTA baselines, demonstrating the effectiveness of statistical adaptation in aligning model predictions with evolving test distributions.

💡 Research Summary

The paper addresses a critical challenge in speech emotion recognition (SER): the degradation of performance when audio‑language models (ALMs) such as CLAP encounter distribution shifts at test time. Existing test‑time adaptation (TTA) techniques either require a few labeled target samples for prompt tuning, rely on gradient‑based updates over batches of unlabeled data, or use heuristic, training‑free methods that still need a buffer of samples. All of these approaches impose constraints that limit their practicality in real‑world, privacy‑sensitive, or low‑latency scenarios where only a single audio utterance is available and model weights cannot be altered.

Emo‑TTA is proposed as a lightweight, training‑free TTA framework that satisfies three desiderata simultaneously: (i) explicit test‑time distribution estimation, (ii) adaptation without any model weight changes, and (iii) operation on a per‑sample basis without storing past inputs. The method treats the CLAP audio embeddings of each emotion class as samples drawn from a multivariate Gaussian distribution with class‑conditional means μ_i and a shared covariance Σ. Class priors π_i are also maintained. Initialization uses the CLAP text encoder to generate semantic prototypes (e.g., “This is a happy sound”) as the initial μ_i, while Σ is set to the identity matrix and π_i to a uniform distribution.

When a new test utterance a_t arrives, its CLAP audio embedding x_t = f(a_t) is processed through an Expectation‑Maximization (EM) step:

• E‑step: Compute soft responsibilities γ_{t,i} = π_i·𝒩(x_t|μ_i,Σ) / Σ_j π_j·𝒩(x_t|μ_j,Σ). These responsibilities represent the posterior probability that the current sample belongs to class i under the current Gaussian model.

• M‑step: Update the parameters incrementally using the new responsibilities and the sample itself:

  • π_i ← (N_i + γ_{t,i}) / n_t,
  • μ_i ← (N_i·μ_i + γ_{t,i}·x_t) / (N_i + γ_{t,i}),
  • Σ ← ((n_t−1)·Σ + Σ_i γ_{t,i}·(x_t−μ_i′)(x_t−μ_i′)^T) / (n_t−1),

where N_i is the effective count of samples assigned to class i so far, and n_t is the total number of observed samples. This online EM formulation allows the model to continuously refine its class statistics without any gradient descent or batch processing.

To mitigate the impact of noisy or highly ambiguous utterances, Emo‑TTA incorporates an entropy‑based confidence weight. The entropy H(a_t) of the CLAP zero‑shot probability distribution is computed, and a weight w(H) = exp(−β·H) (β>0) modulates the influence of the current sample on the updates. High‑entropy (uncertain) samples thus contribute less, stabilizing early adaptation.

Finally, predictions are obtained by fusing two sources of information:

1. The log‑likelihood from the updated Gaussian model, expressed as w_i·F + b_i where w_i = Σ^{-1}μ_i and b_i = log π_i − ½ μ_i^T Σ^{-1} μ_i.
2. The original CLAP cosine similarity logits T_i·F (where T_i = g(t_i) is the text prototype).

The combined score is α·(w_i·F + b_i) + (1−α)·(T_i·F), with α set to 0.2 in experiments. This hybrid scoring leverages both the semantic alignment provided by CLAP and the dynamically adapted statistical model.

Experimental Evaluation

The authors evaluate Emo‑TTA on six out‑of‑domain SER datasets: IEMOCAP, MELD, RAVDESS, TESS, SAVEE, and CREMA‑D. Two CLAP backbones are used: PANN‑14 and HTSAT. Baselines include prompt‑learning methods (CoOp, CoCoOp), gradient‑based TTA (Tref‑Adapter, TPT), and training‑free approaches (MT‑A, ZERO). All baselines are adapted to use CLAP encoders for a fair comparison.

Key results:

• With CLAP‑PANN‑14, Emo‑TTA achieves an average accuracy of 38.02 %, surpassing the zero‑shot CLAP baseline (31.37 %) by 6.65 % points and beating the strongest prior method (Tref‑Adapter, 36.11 %) by 1.91 % points.
• With CLAP‑HTSAT, Emo‑TTA reaches 40.47 % average accuracy, again leading all baselines (the next best is MT‑A at 36.04 %).
• Across 12 backbone‑dataset combinations, Emo‑TTA attains the highest score in 10 cases, demonstrating robustness to diverse acoustic conditions, speaker variations, and label taxonomies.

Ablation Studies

The authors conduct three ablations:

1. Static class means – freezing μ_i to the initial text prototypes reduces accuracy by ~3–5 % points, confirming the necessity of adapting means.
2. No covariance update – keeping Σ fixed to the identity matrix leads to a similar drop, highlighting the benefit of modeling feature correlations.
3. No entropy weighting / no ALM priors – removing the confidence weighting or the CLAP prior degrades performance, indicating that both the prior information and uncertainty handling are crucial for stable adaptation.

Discussion and Limitations

Emo‑TTA’s primary strength lies in its simplicity and practicality: it requires no additional training, no gradient computation, and operates on a single utterance at a time, making it suitable for on‑device or privacy‑preserving deployments. However, the method assumes that class‑conditional embeddings follow a Gaussian distribution with a shared covariance, which may be restrictive for highly non‑linear emotional manifolds. Future work could explore mixture‑of‑Gaussians, non‑parametric density estimation, or Bayesian deep learning extensions to capture more complex structures.

Conclusion

The paper introduces Emo‑TTA, an EM‑based, training‑free test‑time adaptation framework for audio‑language models in speech emotion recognition. By incrementally updating class‑conditional Gaussian statistics and integrating entropy‑aware confidence, Emo‑TTA aligns model predictions with the evolving test distribution without any weight updates or batch processing. Extensive experiments across six OOD SER benchmarks demonstrate consistent and significant accuracy gains over a wide range of strong baselines, establishing Emo‑TTA as a compelling solution for real‑world, low‑latency SER applications.