Privacy-preserving Speech Emotion Recognition through Semi-Supervised Federated Learning

Vasileios Tsouvalas; Tanir Ozcelebi; Nirvana Meratnia

FedSEREmotionAware @ IEEE PerCom · 2022

Privacy-preserving Speech Emotion Recognition through Semi-Supervised Federated Learning

Learn speech emotions from audio kept on each device

Vasileios Tsouvalas Tanir Ozcelebi Nirvana Meratnia

Paper Code Poster

At a glance

Emotion labels are costly, while speech recordings are sensitive. This study combines spectral, temporal and channel attention with local self-training, using unlabeled speech while sharing model parameters across devices.

Inside FedSER

Represent the speechLog-mel features expose time and frequency structure. Four convolutional blocks combine temporal and spectral filters into learned feature channels.

Reweight useful featuresTemporal, spectral and channel attention maps are combined and normalized. Their weights multiply the CNN features before emotion classification.

Account for client participationThe confidence schedule uses global progress and each client’s participation history. Local predictions must clear the client’s threshold to supply pseudo-labels.

Learn locally, aggregate globallyKnown labels and accepted pseudo-labels contribute jointly to local learning. The server averages model parameters without requiring its own speech dataset.

What this adds

01
Attention across three axes reweights the local model’s speech features.
02
Participation-aware self-training uses confident predictions on unlabeled recordings.
03
Four IEMOCAP partitions separate speaker overlap from how speech is distributed across clients.

Abstract

Speech Emotion Recognition (SER) refers to the recognition of human emotions from natural speech. If done accurately, it can offer a number of benefits in building human-centered context-aware intelligent systems. Existing SER approaches are largely centralized, without considering users' privacy. Federated Learning (FL) is a distributed machine learning paradigm dealing with decentralization of privacy-sensitive personal data. In this paper, we present a privacy-preserving and data-efficient SER approach by utilizing the concept of FL. To the best of our knowledge, this is the first federated SER approach, which utilizes self-training learning in conjunction with federated learning to exploit both labeled and unlabeled on-device data. Our experimental evaluations on the IEMOCAP dataset shows that our federated approach can learn generalizable SER models even under low availability of data labels and highly non-i.i.d. distributions. We show that our approach with as few as 10% labeled data, on average, can improve the recognition rate by 8.67% compared to the fully-supervised federated counterparts.

FedSER

Speech representation

The method uses 64-bin log-mel features and four convolutional blocks. Predictions over non-overlapping utterance segments are averaged.

§III, public preprint p. 3. Method text specifies two-second segments; Figure 1’s one-second label is inconsistent.

Spectral, temporal and channel attention

Pooling and convolution produce spectral/temporal maps; pooled channels pass through a small perceptron. Their combined, softmax-scaled weights multiply the learned features.

§III and Figure 1. The illustration explains feature weighting, not a validated interpretation of human emotion.

Joint self-training

Known labels supply one cross-entropy term; confident predicted classes supply another. β controls the unlabeled contribution.

ℒ_{k} = ℒ_{s} + β ℒ_{u}

Eq. 1 and Algorithm 1. Temperature 2 is used when constructing pseudo-labels.

Participation-aware confidence and FedAvg

The threshold depends on global progress and local participation. Clients send updated parameters for data-weighted FedAvg aggregation.

§III–IV. The prose describes a threshold increasing from 0.5 to 0.9; printed Eq. 3 conflicts with that description, so no corrected schedule is inferred here.

Method details in the paper ↗

Original paper figures & sources

An attention-based convolutional model analyzes audio spectrograms while federated clients train using labeled inputs and confidence-filtered pseudo-labels from unlabeled inputs. — Spectral, temporal, and channel attention help the local model represent speech. Federated self-training adds supervision from unlabeled audio, and clients share model updates with the server. Figure 1 · public preprint, p. 3

Evaluation

Results

Results from arXiv:2202.02611v1 ↗

Self-training with 10% labels

Self-training improves every equal-label comparison but remains below the fully labeled references.

Unweighted accuracy (%)

Higher is better

Supervised · 10% labelsSelf-training · 10% labelsSupervised · 100% labels

0100

Unseen speakers · random

46.72

53.94

60.52

Unseen speakers · per speaker

46.55

56.83

59.55

Overlapping speakers · random

51.58

59.04

64.78

Overlapping speakers · per speaker

49.97

59.72

62.79

Experiment details

IEMOCAP improvised subset; neutral, happy, sad and angry. Mean unweighted accuracy (UA, %) over five runs per split; higher is better; no dispersion reported. Random partitions use 10 clients and per-speaker partitions use 8.

Unseen-speaker results use leave-one-session-out evaluation; overlapping-speaker results use 80:20 splits with five-fold cross-validation. The latter is an easier generalization setting.

Unlabeled speech helps with 10% labels. IEMOCAP improvised subset; neutral, happy, sad and angry. Mean unweighted accuracy (UA, %) over five runs per split; higher is better; no dispersion reported. Random partitions use 10 clients and per-speaker partitions use 8.
Partition	Supervised · 10% labels	Self-training · 10% labels	Supervised · 100% labels
Unseen speakers · random	46.72	53.94	60.52
Unseen speakers · per speaker	46.55	56.83	59.55
Overlapping speakers · random	51.58	59.04	64.78
Overlapping speakers · per speaker	49.97	59.72	62.79

Table III · page 5 ↗

Centralized and federated references

Federated training reduces the same architecture’s UA by 2.06 points for unseen speakers and 4.26 points with overlapping speakers. The CNN Transformer is a literature reference.

Unweighted accuracy (%)

Higher is better

Unseen speakers (%)Overlapping speakers (%)

0100

CNN Transformer (cited baseline)

64.79

70.05

Proposed model · centralized

62.58

69.04

Proposed model · federated

60.52

64.78

Experiment details

Same four-emotion IEMOCAP subset; fully supervised random partitions. Mean UA (%), higher is better. The paper’s federated model uses 10 clients, 100 rounds, 80% participation and one local epoch; no dispersion reported.

The CNN Transformer is a cited literature reference, not an ablation of the proposed architecture. Centralized and federated results should be compared within each speaker protocol.

Centralized and federated references. Same four-emotion IEMOCAP subset; fully supervised random partitions. Mean UA (%), higher is better. The paper’s federated model uses 10 clients, 100 rounds, 80% participation and one local epoch; no dispersion reported.
Model	Unseen speakers (%)	Overlapping speakers (%)
CNN Transformer (cited baseline)	64.79	70.05
Proposed model · centralized	62.58	69.04
Proposed model · federated	60.52	64.78

Table II · page 5 ↗

Study scope & sources

Results use the IEMOCAP improvised subset and four emotions; they do not establish cross-corpus or cross-language transfer.
FedSER is a project shorthand for the EmotionAware / IEEE PerCom Workshops 2022 paper. Results and figures are sourced from its public preprint.

Data: IEMOCAP improvised speech only, four emotion categories; 16 kHz audio and two-second segments in the method text.
Models: Four-block convolutional model with spectral, temporal and channel attention; predictions averaged across non-overlapping utterance segments.
Compared with: Supervised FedAvg using the identical 10% labeled subset, fully labeled FedAvg, and centralized attention-based SER references.
Protocol: Flower simulation; leave-one-session-out for unseen speakers and five-fold 80:20 splits for overlapping speakers; random versus per-speaker client partitions.

Scope & limitations

Evidence is limited to one dataset’s improvised subset and four emotions; cross-language, cross-corpus and natural deployment generalization are untested.
Keeping recordings on devices reduces raw-data sharing but does not establish differential privacy or resistance to information leakage through updates.
The comparison reports means without dispersion, and per-speaker and random partitions also differ in client count.
The work simulates clients and does not provide real-device training energy or latency measurements.

arXiv:2202.02611v1, 5 February 2022; 6-page public preprint of the EmotionAware / PerCom Workshops paper, no appendix. ↗

Reference

Cite this work

BibTeX

Download .bib

@inproceedings{tsouvalas2022fedser,
  author = {Tsouvalas, Vasileios and Ozcelebi, Tanir and Meratnia, Nirvana},
  title = {Privacy-preserving Speech Emotion Recognition through Semi-Supervised Federated Learning},
  booktitle = {2022 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops)},
  year = {2022},
  pages = {359--364},
  note = {EmotionAware workshop},
  doi = {10.1109/PerComWorkshops53856.2022.9767445},
  url = {https://doi.org/10.1109/PerComWorkshops53856.2022.9767445}
}

Keep exploring

FedSTAR

Turn unlabeled audio into shared learning

FedCode

Exchange codebooks to make federated communication smaller