Federated Self-training for Semi-supervised Audio Recognition

Vasileios Tsouvalas; Aaqib Saeed; Tanir Ozcelebi

FedSTARACM TECS · 2022

Federated Self-training for Semi-supervised Audio Recognition

Turn unlabeled audio into shared learning

Vasileios Tsouvalas Aaqib Saeed Tanir Ozcelebi

Paper Code Poster

At a glance

Most device audio has no labels. FedSTAR turns selected local predictions into training targets, combining them with known labels before sharing model updates. Optional public-audio pretraining supplies a stronger starting point.

Inside FedSTAR

Prepare a starting pointOptionally pretrain on public FSD-50K: bring segments of the same clip together and distinguish other clips. This happens before downstream federated rounds.

Use both local batchesA selected client receives the global model and forms equally sized labeled and unlabeled batches. Both contribute to local learning.

Filter predicted targetsTemperature scaling softens confidence without changing the top class. Only predictions above a threshold become pseudo-labels; that threshold rises as training progresses.

Combine the learning signalsCompute both cross-entropy terms, weight the unlabeled term and add them before backpropagation. Teacher and student describe roles within self-training.

Share learned parametersFedAvg combines participating clients’ parameters by their data weights. The updated global model starts the next round; downstream recordings remain local.

What this adds

01
Local self-training brings unlabeled audio into the same update as labeled examples.
02
Temperature and confidence filtering limit which predicted targets influence learning.
03
Optional contrastive initialization improves early convergence; three audio tasks test label scarcity and client variation.

Abstract

Federated Learning is a distributed machine learning paradigm dealing with decentralized and personal datasets. Since data reside on devices such as smartphones and virtual assistants, labeling is entrusted to the clients or labels are extracted in an automated way. Specifically, in the case of audio data, acquiring semantic annotations can be prohibitively expensive and time-consuming. As a result, an abundance of audio data remains unlabeled and unexploited on users’ devices. Most existing federated learning approaches focus on supervised learning without harnessing the unlabeled data. In this work, we study the problem of semi-supervised learning of audio models via self-training in conjunction with federated learning. We propose FedSTAR to exploit large-scale on-device unlabeled data to improve the generalization of audio recognition models. We further demonstrate that self-supervised pre-trained models can accelerate the training of on-device models, significantly improving convergence within fewer training rounds. We conduct experiments on diverse public audio classification datasets and investigate the performance of our models under varying percentages of labeled and unlabeled data. Notably, we show that with as little as 3% labeled data available, FedSTAR on average can improve the recognition rate by 13.28% compared to the fully supervised federated model.

FedSTAR

Joint local objective

The client learns from known labels and retained pseudo-labels in one update. β = 0.5 weights the unlabeled term in the reported experiments.

ℒ_{k} = ℒ_{s} + β ℒ_{u}

ACM TECS §4.1–4.3, Eqs. 3 and 7. Both losses are computed before backpropagation.

Confidence-filtered pseudo-labels

Apply softmax to temperature-scaled logits, take the most likely class and retain predictions above τ. Temperature is 4; the cosine threshold increases from 0.5 to 0.9.

\hat{y} = \underset{c}{argmax} \frac{e^{z_{c} / T}}{\sum_{j} e^{z_{j} / T}}

Eq. 5 and Table 2. Confidence filtering reduces exposure to uncertain targets; it does not guarantee correct labels.

Ordinary federated aggregation

A sampled set of clients trains locally and returns model parameters. FedAvg forms a data-weighted average; the self-training change is inside the client update.

Algorithm 1, PDF p. 12. Experiments simulate clients with the Flower framework.

Optional public-data initialization

Contrastive pretraining on FSD-50K supplies initial encoder weights. The projection layer is discarded before downstream learning.

§4.4 and Figure 6. Better performance after 10 rounds is distinct from final accuracy after 100 rounds.

Method details in the paper ↗

Original paper figures & sources

Federated server coordinates several clients, each combining labeled audio and confidence-filtered pseudo-labels from unlabeled audio in local self-training. — Each device learns from the labels it has and the confident predictions it can make. The server combines local model updates into a shared audio recognition model. Figure 1 · ACM TECS article, p. 74:3

Evaluation

Results

Results from the source manuscript ↗

Learning with 3% labels

With 15 clients and 3% labels, self-training improves all three tasks; the gain varies substantially by task.

Accuracy (%)

Higher is better

Supervised FL (%)FedSTAR (%)

0100

Ambient Context

33.03 ± 0.002

49.54 ± 0.005

Speech Commands

62.98 ± 0.003

86.82 ± 0.006

VoxForge

54.26 ± 0.002

55.82 ± 0.011

Experiment details

15 clients; 3% labels, remaining clips available unlabeled; 100 rounds, one local epoch, 80% participation, data-distribution variance 25%, unlabeled-loss weight 0.5. Mean test accuracy (%) over three runs; higher is better.

The appendix prints ± values and calls them variance across experiments; they are retained verbatim, with no conversion to standard deviations or confidence intervals. These rows share a client count and label budget, but represent different tasks; their accuracies should not be pooled.

Learning with 3% labels. 15 clients; 3% labels, remaining clips available unlabeled; 100 rounds, one local epoch, 80% participation, data-distribution variance 25%, unlabeled-loss weight 0.5. Mean test accuracy (%) over three runs; higher is better.
Dataset	Supervised FL (%)	FedSTAR (%)
Ambient Context	33.03 ± 0.002	49.54 ± 0.005
Speech Commands	62.98 ± 0.003	86.82 ± 0.006
VoxForge	54.26 ± 0.002	55.82 ± 0.011

Appendix Table 9 · PDF page 25 (74:24) ↗

When labeled classes are missing

Unlabeled class coverage supplies information missing from each client’s labeled subset in this specific Speech Commands setup.

Accuracy (%)

Higher is better

Supervised FL (%)FedSTAR (%)

0100

9.83 ± 0.017

79.08 ± 0.026

25%

10.54 ± 0.016

79.05 ± 0.052

50%

8.44 ± 0.001

78.14 ± 0.021

Experiment details

Speech Commands; 15 clients, 3% labels, mean of three labeled classes per client out of 12; unlabeled clips are randomly distributed and may cover all classes. 100 rounds, one local epoch, 80% participation. Mean test accuracy (%), higher is better.

The appendix prints ± values and calls them variance across experiments; they are retained verbatim, with no conversion to standard deviations or confidence intervals. This does not test the case where both labeled and unlabeled client data lack the same classes.

Labeled classes missing locally. Speech Commands; 15 clients, 3% labels, mean of three labeled classes per client out of 12; unlabeled clips are randomly distributed and may cover all classes. 100 rounds, one local epoch, 80% participation. Mean test accuracy (%), higher is better.
Class-distribution variance	Supervised FL (%)	FedSTAR (%)
0%	9.83 ± 0.017	79.08 ± 0.026
25%	10.54 ± 0.016	79.05 ± 0.052
50%	8.44 ± 0.001	78.14 ± 0.021

Appendix Table 8 · PDF page 24 (74:23) ↗

Pretraining at 100 rounds

At 50% unlabeled use and 100 rounds, pretraining helps with 20% and 50% labels, but not with the two smaller label budgets.

Accuracy (%)

Higher is better

Random initialization (%)FSD-50K pretraining (%)

0100

85.40 ± 0.008

85.17 ± 0.001

88.52 ± 0.005

88.28 ± 0.002

20%

92.24 ± 0.012

93.67 ± 0.005

50%

92.26 ± 0.005

93.83 ± 0.003

Experiment details

Speech Commands; 15 clients, 50% of the remaining unlabeled pool used; label budget varies by row. 100 rounds, one local epoch, 80% participation. Mean test accuracy (%) over three runs; higher is better.

The appendix prints ± values and calls them variance across experiments; they are retained verbatim, with no conversion to standard deviations or confidence intervals. This endpoint comparison is distinct from Figure 6’s early-convergence experiment at 10 rounds.

Pretraining after 100 rounds. Speech Commands; 15 clients, 50% of the remaining unlabeled pool used; label budget varies by row. 100 rounds, one local epoch, 80% participation. Mean test accuracy (%) over three runs; higher is better.
Labeled data	Random initialization (%)	FSD-50K pretraining (%)
3%	85.40 ± 0.008	85.17 ± 0.001
5%	88.52 ± 0.005	88.28 ± 0.002
20%	92.24 ± 0.012	93.67 ± 0.005
50%	92.26 ± 0.005	93.83 ± 0.003

Appendix Table 7 · PDF page 24 (74:23) ↗

Study scope & sources

Evidence is from the 2022 ACM TECS journal article; the related ICASSP paper remains a separate publication.
Client training is simulated. Keeping raw audio local establishes data locality, not differential privacy.

Data: Ambient Acoustic Contexts (10 events), Speech Commands v2 (12 classes), VoxForge (6 languages); FSD-50K for optional contrastive pretraining.
Models: Four-block convolutional audio network using log-mel inputs, group normalization and spatial dropout.
Compared with: Centralized supervised training and supervised FedAvg with matching local labeled subsets; random versus pretrained FedSTAR.
Protocol: Flower simulations with 5–30 clients, 20–80% participation, 1–4 local epochs and three full runs per setting.

Scope & limitations

Self-training needs some reliable labels and assumes local labeled and unlabeled audio come from the same distribution.
The strongest class-skew gains use unlabeled data that may contain all classes even where labeled data do not.
More local training can harm the 3% label setting through overfitting or incorrect pseudo-labels; benefits are not universal across configurations.
Experiments simulate federated clients. Real-device energy, system heterogeneity and formal differential privacy are not established.

ACM TECS 21(6), Article 74, 2022; final publisher PDF with repository cover and appendix (27 PDF pages). ↗

Reference

Cite this work

BibTeX

Download .bib

@article{tsouvalas2022fedstar,
  author = {Tsouvalas, Vasileios and Saeed, Aaqib and Ozcelebi, Tanir},
  title = {Federated Self-training for Semi-supervised Audio Recognition},
  journal = {ACM Transactions on Embedded Computing Systems},
  year = {2022},
  volume = {21},
  number = {6},
  articleno = {74},
  numpages = {26},
  doi = {10.1145/3520128},
  url = {https://doi.org/10.1145/3520128}
}

Keep exploring

FedSER

Learn speech emotions from audio kept on each device

FedCode

Exchange codebooks to make federated communication smaller

Federated Self-training for Semi-supervised Audio Recognition

What this adds

Scope & limitations

Cite this work

Related publication

Keep exploring