Learn speech emotions from audio kept on each device
Emotion labels are costly, while speech recordings are sensitive. This study combines spectral, temporal and channel attention with local self-training, using unlabeled speech while sharing model parameters across devices.
Attention across three axes reweights the local model’s speech features.
Participation-aware self-training uses confident predictions on unlabeled recordings.
Four IEMOCAP partitions separate speaker overlap from how speech is distributed across clients.
Speech Emotion Recognition (SER) refers to the recognition of human emotions from natural speech. If done accurately, it can offer a number of benefits in building human-centered context-aware intelligent systems. Existing SER approaches are largely centralized, without considering users' privacy. Federated Learning (FL) is a distributed machine learning paradigm dealing with decentralization of privacy-sensitive personal data. In this paper, we present a privacy-preserving and data-efficient SER approach by utilizing the concept of FL. To the best of our knowledge, this is the first federated SER approach, which utilizes self-training learning in conjunction with federated learning to exploit both labeled and unlabeled on-device data. Our experimental evaluations on the IEMOCAP dataset shows that our federated approach can learn generalizable SER models even under low availability of data labels and highly non-i.i.d. distributions. We show that our approach with as few as 10% labeled data, on average, can improve the recognition rate by 8.67% compared to the fully-supervised federated counterparts.
Read more ↗The method uses 64-bin log-mel features and four convolutional blocks. Predictions over non-overlapping utterance segments are averaged.
§III, public preprint p. 3. Method text specifies two-second segments; Figure 1’s one-second label is inconsistent.
Pooling and convolution produce spectral/temporal maps; pooled channels pass through a small perceptron. Their combined, softmax-scaled weights multiply the learned features.
§III and Figure 1. The illustration explains feature weighting, not a validated interpretation of human emotion.
Known labels supply one cross-entropy term; confident predicted classes supply another. β controls the unlabeled contribution.
Eq. 1 and Algorithm 1. Temperature 2 is used when constructing pseudo-labels.
The threshold depends on global progress and local participation. Clients send updated parameters for data-weighted FedAvg aggregation.
§III–IV. The prose describes a threshold increasing from 0.5 to 0.9; printed Eq. 3 conflicts with that description, so no corrected schedule is inferred here.
IEMOCAP · four emotions · 8–10 clients · mean of five runs
@inproceedings{tsouvalas2022fedser,
author = {Tsouvalas, Vasileios and Ozcelebi, Tanir and Meratnia, Nirvana},
title = {Privacy-preserving Speech Emotion Recognition through Semi-Supervised Federated Learning},
booktitle = {2022 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops)},
year = {2022},
pages = {359--364},
note = {EmotionAware workshop},
doi = {10.1109/PerComWorkshops53856.2022.9767445},
url = {https://doi.org/10.1109/PerComWorkshops53856.2022.9767445}
}