Why emotion models struggle beyond their training domain
An emotion model can fit one speech corpus and fail on another. DA-MLDG rehearses source-domain shifts during training and weights them with an energy-gap estimate. The study finds mixed generalization gains and exposes limits of the representations and benchmarks.
Domain-shift-aware meta-learning weights the loss on a held-out source subset using an energy gap.
Seven cross-corpus scenarios make both improvements and failure cases visible.
Representation analysis compares log-mel inputs with TRILLsson embeddings without claiming domain shift is solved.
Speech Emotion Recognition (SER) refers to the recognition of human emotions from natural speech, vital for building human-centered context-aware intelligent systems. Here, domain shift, where models’ trained on one domain exhibit performance degradation when exposed to an unseen domain with different statistics, is a major limiting factor in SER applicability, as models have a strong dependence on speakers and languages characteristics used during training. Meta-Learning for Domain Generalization (MLDG) has shown great success in improving models’ generalization capacity and alleviate the domain shift problem in the vision domain; yet, its’ efficacy on SER remains largely unexplored. In this work, we propose a “domain-shift aware” MLDG approach to learn generalizable models across multiple domains in SER. Based on our extensive evaluation, we identify a number of pitfalls that contribute to poor models’ DG ability, and demonstrate that log-mel spectrograms representations lack distinct features required for MLDG in SER. We further explore the use of appropriate features to achieve DG in SER as to provide insides to future research directions for DG in SER.
Read more ↗Partition source domains into disjoint sets S′ and V at each iteration. A gradient step on S′ produces adapted parameters θ′.
§III, Eq. 2 and Algorithm 1. V is a source meta-test subset, not the final target test corpus.
Compute each subset’s average log-sum-exp of logits with θ′ and temperature 1. The absolute gap supplies γ, the adaptive weight.
Eq. 3, printed p. 423. The score is proposed as a proxy for domain shift, not a measured distance between speech distributions.
Minimize source training loss plus γ times meta-test loss after adaptation. The outer optimization accounts for how θ′ depends on θ.
Eq. 2. The formula presents the stated objective rather than Algorithm 1’s outer-update line, which omits an explicit gradient symbol.
Compare ResNet-18 with log-mel inputs against a variant using TRILLsson embeddings. Seven source/target scenarios cover same-language and cross-language transfer.
Tables I, III and IV; ten trials per setting. The target corpora remain unseen during training.
Results from the source manuscript ↗
DA-MLDG beats Deep-All in four of seven scenarios and leads all four methods in only two. The exact table retains RSC and MLDG comparisons.
Five-emotion SER with VoxCeleb-pretrained ResNet-18 and log-mel inputs; source and target corpora are defined in Table I. Mean target accuracy (%) over ten trials; higher is better. No dispersion is reported for these results.
Scenarios use different source corpora, so ShEMO appears twice with different training conditions. ShEMO lacks the disgust class. The small 0.05-point lead in scenario 4 has no reported uncertainty.
| Scenario · target | Deep-All (%) | RSC (%) | MLDG (%) | DA-MLDG (%) |
|---|---|---|---|---|
| 1 · IEMOCAP | 26.95 | 32.36 | 27.61 | 31.35 |
| 2 · RAVDESS | 43.40 | 44.78 | 38.86 | 39.41 |
| 3 · TESS | 47.03 | 43.77 | 46.29 | 46.80 |
| 4 · CaFE | 43.16 | 39.97 | 42.57 | 43.21 |
| 5 · AESDD | 60.65 | 56.69 | 60.45 | 62.26 |
| 6 · ShEMO | 31.54 | 40.70 | 37.87 | 38.79 |
| 7 · ShEMO | 40.19 | 40.27 | 34.51 | 37.11 |
TRILLsson improves DA-MLDG over its log-mel version in six scenarios, yet it beats the supervised embedding baseline in only three.
The same seven SER scenarios using TRILLsson v3 EfficientNetv2-B3 embeddings as inputs to ResNet-18. Mean target accuracy (%) over ten trials; higher is better. No dispersion is reported.
These inputs add external pretraining. The results still show substantial target-domain errors and do not demonstrate practical readiness.
| Scenario · target | Deep-All (%) | DA-MLDG (%) |
|---|---|---|
| 1 · IEMOCAP | 28.57 | 32.11 |
| 2 · RAVDESS | 42.61 | 39.82 |
| 3 · TESS | 50.16 | 48.46 |
| 4 · CaFE | 45.09 | 43.29 |
| 5 · AESDD | 61.17 | 64.11 |
| 6 · ShEMO | 33.16 | 40.22 |
| 7 · ShEMO | 41.15 | 36.97 |
@inproceedings{gandhi2023damldg,
author = {Gandhi, Raeshak King and Tsouvalas, Vasileios and Meratnia, Nirvana},
title = {On efficacy of Meta-Learning for Domain Generalization in Speech Emotion Recognition},
booktitle = {2023 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops)},
year = {2023},
pages = {421--426},
note = {PerFail 2023: Second International Workshop on Negative Results in Pervasive Computing},
doi = {10.1109/PerComWorkshops56833.2023.10150265},
url = {https://doi.org/10.1109/PerComWorkshops56833.2023.10150265}
}