Communication-Efficient Federated Learning through Adaptive Weight Clustering and Server-Side Distillation

Vasileios Tsouvalas; Aaqib Saeed; Tanir Ozcelebi; Nirvana Meratnia

FedCompressIEEE ICASSP · 2024

Communication-Efficient Federated Learning through Adaptive Weight Clustering and Server-Side Distillation

Compress both directions of federated training.

Vasileios Tsouvalas Aaqib Saeed Tanir Ozcelebi Nirvana Meratnia

Paper Code Poster

At a glance

Compress both directions of federated training. Clustering makes client uploads compact, but averaging can undo that structure. FedCompress restores it with server-side distillation and adjusts cluster capacity using an unlabeled representation score.

Inside FedCompress

Train weights around shared valuesAfter a local warm-up, clients combine task learning with a clustering penalty. Many weights share a small set of learned centroids, making the upload compact.

Averaging can break compressionClients learn different centroids and assignments. Standard averaging can create more distinct values than either client used; compact uploads therefore do not ensure a compact download.

Distill a clustered global studentThe aggregated model teaches a clustered student on unlabeled server data. Matching predictions while enforcing clustering restores a compact model for the next download.

Grow capacity when progress stallsClients compute effective rank from unlabeled embeddings. The server combines these scores and monitors their moving average, increasing cluster capacity when progress stalls.

What this adds

01
Dual compression: clustered local training plus server self-compression after aggregation.
02
Adaptive capacity: use embedding effective rank to guide the number of shared weight values.
03
Vision and audio evidence: report communication, model size and device inference separately.

Abstract

Federated Learning (FL) is a promising technique for the collaborative training of deep neural networks across multiple devices while preserving data privacy. Despite its potential benefits, FL is hindered by excessive communication costs due to repeated server-client communication during training. To address this challenge, model compression techniques, such as sparsification and weight clustering are applied, which often require modifying the underlying model aggregation schemes or involve cumbersome hyperparameter tuning, with the latter not only adjusts the model's compression rate but also limits model's potential for continuous improvement over growing data. In this paper, we propose FedCompress, a novel approach that combines dynamic weight clustering and server-side knowledge distillation to reduce communication costs while learning highly generalizable models. Through a comprehensive evaluation on diverse public datasets, we demonstrate the efficacy of our approach compared to baselines in terms of communication costs and inference speed.

FedCompress

Learn task and compression together

Each client minimizes cross-entropy plus a weight-clustering penalty. The local warm-up uses β=0 before enabling clustering with β=1.

L_{k} = L_{ce} + β L_{wc} (θ_{k}, μ, C)

Equation 1, PDF page 3. μ denotes learnable centroids; C is their count.

Restore clusters after aggregation

Keep the underlying aggregation rule. Then optimize a clustered student to match the temperature-scaled teacher outputs on unlabeled auxiliary data held by the server.

L_{s} = L_{KL} (p_{T} ∥ p) + β L_{wc}

Equation 2, PDF page 4, with model arguments abbreviated. The KL term includes the paper’s temperature scaling.

Measure representation diversity

Normalize the singular values of local embeddings and compute their effective rank. This is an unlabeled proxy for representation quality, not a generalization guarantee.

E = \exp (- \sum_{j} r_{j} \log r_{j})

Dynamic weight-clustering section, PDF page 4. r is the normalized singular-value spectrum; the implementation description adds a small stability constant.

Use the score to adjust capacity

Start with a small cluster count and increase it when score progress stalls. The prose uses a three-round window and patience, with an optional maximum cluster budget.

PDF page 4. This describes the intended schedule; Appendix A contains notation differences and is not an independently verified implementation.

Method details in the paper ↗

Original paper figures & sources

FedCompress diagram showing local training with clustered weights, server aggregation, self-compression through teacher-student distillation on out-of-distribution data, and dynamic adjustment of cluster counts. — Weight clustering compresses client uploads. After aggregation, server-side distillation restores a clustered global model for the return trip, while a representation-quality score adjusts the number of clusters. Figure 1 in the public preprint

Evaluation

Results

Results from arXiv:2401.14211v3 ↗

Savings and accuracy

3.80–5.41× less communication accompanies accuracy losses of 0.31–1.88 percentage points across the five datasets.

Communication reduction (×)

Higher is better

CIFAR-10

4.53

CIFAR-100

3.80

PathMNIST

4.79

SpeechCommands

5.04

VoxForge

5.41

Accuracy change (pp)

-20

CIFAR-10

−1.83

CIFAR-100

−1.88

PathMNIST

−1.72

SpeechCommands

−0.42

VoxForge

−0.31

Experiment details

20 clients, 20 rounds, ten client epochs and ten server self-compression epochs per round. Accuracy is averaged over two trials. Δ accuracy is percentage points relative to FedAvg; CCR and MCR are reduction factors, higher is better.

No variance or confidence interval is tabulated for the two trials. The source’s narrative averages do not consistently match Table 1, so individual entries are used here.

Communication savings and accuracy cost. 20 clients, 20 rounds, ten client epochs and ten server self-compression epochs per round. Accuracy is averaged over two trials. Δ accuracy is percentage points relative to FedAvg; CCR and MCR are reduction factors, higher is better.
Dataset	FedAvg accuracy (%)	FedCompress Δ accuracy (pp)	Communication reduction (×)	Model compression (×)
CIFAR-10	86.26	−1.83	4.53	5.18
CIFAR-100	60.68	−1.88	3.80	3.93
PathMNIST	88.22	−1.72	4.79	5.27
SpeechCommands	95.75	−0.42	5.04	5.09
VoxForge	81.05	−0.31	5.41	5.64

Table 1 · PDF page 5 ↗

Server compression ablation

Restoring global clusters raises the reported communication reduction from 1.02× to 3.80× on CIFAR-100, with additional server training.

Communication reduction (×)

Higher is better

FedZip

1.94

FedCompress without server self-compression

1.02

FedCompress

3.80

Experiment details

CIFAR-100 with ResNet-20 under the same 20-client, 20-round protocol. Accuracy differences are percentage points relative to FedAvg; communication and model-size reductions are factors relative to FedAvg.

Self-compression uses ten server epochs per round on auxiliary unlabeled data; the communication benefit is not a claim of free computation.

Why recompress at the server?. CIFAR-100 with ResNet-20 under the same 20-client, 20-round protocol. Accuracy differences are percentage points relative to FedAvg; communication and model-size reductions are factors relative to FedAvg.
Method	Δ accuracy (pp)	Communication reduction (×)	Model compression (×)
FedZip	−2.57	1.94	2.11
FedCompress without server self-compression	−2.67	1.02	1.62
FedCompress	−1.88	3.80	3.93

Table 1 · CIFAR-100 · PDF page 5 ↗

Device inference speedup

Relative inference speedups are 1.103–1.152× for float32 and 1.161–1.248× for uint8. Each uses its own same-device, same-model FedAvg reference.

Inference speedup (×)

Higher is better

Float32 speedup (×)Uint8 speedup (×)

ResNet-20 · Pixel 6

1.103

1.165

ResNet-20 · Jetson Nano

1.127

1.169

ResNet-20 · Coral TPU

1.113

1.191

MobileNet · Pixel 6

1.114

1.248

MobileNet · Jetson Nano

1.137

1.161

MobileNet · Coral TPU

1.152

1.194

Experiment details

ResNet-20 on CIFAR-10 and MobileNet on SpeechCommands. Each factor compares inference time with that device/model’s FedAvg reference; higher is better. Float32 and uint8 are separate model formats.

Absolute latency, variability and energy measurements are not supplied. These factors do not rank device speed or measure federated training time.

Measured inference acceleration. ResNet-20 on CIFAR-10 and MobileNet on SpeechCommands. Each factor compares inference time with that device/model’s FedAvg reference; higher is better. Float32 and uint8 are separate model formats.
Model · device	Float32 speedup (×)	Uint8 speedup (×)
ResNet-20 · Pixel 6	1.103	1.165
ResNet-20 · Jetson Nano	1.127	1.169
ResNet-20 · Coral TPU	1.113	1.191
MobileNet · Pixel 6	1.114	1.248
MobileNet · Jetson Nano	1.137	1.161
MobileNet · Coral TPU	1.152	1.194

Table 2 · PDF page 6 ↗

Study scope & sources

20 clients and 20 rounds; small vision and audio models.
Two trials without tabulated uncertainty; individual table entries take precedence over inconsistent prose averages.

Data: CIFAR-10/100 and PathMNIST for vision; SpeechCommands and VoxForge for audio.
Models: ResNet-20 for vision; MobileNet for audio; standard FedAvg aggregation.
Auxiliary server data: StyleGAN (Oriented) for vision and LibriSpeech for audio; unlabeled data supports teacher–student self-compression.
Compared with: FedAvg, FedZip with 15 clusters, and FedCompress without server self-compression; inference on Pixel 6, Jetson Nano and Coral TPU.

Scope & limitations

The main training study uses 20 clients and 20 rounds; it does not establish scalability to foundation models or large client populations.
Server self-compression requires auxiliary data and additional optimization; sensitivity to the auxiliary distribution is not broadly quantified.
Accuracy averages use two trials without tabulated uncertainty, and some prose averages conflict with the reported table entries.
Inference acceleration is relative and modest; the paper does not directly measure end-to-end training latency or energy savings.

arXiv:2401.14211v3, 25 February 2024; 9 pages including Appendix A (Algorithm 1) ↗

Reference

Cite this work

BibTeX

Download .bib

@inproceedings{tsouvalas2024fedcompress,
  title = {Communication-Efficient Federated Learning through Adaptive Weight Clustering and Server-Side Distillation},
  author = {Vasileios Tsouvalas and Aaqib Saeed and Tanir Ozcelebi and Nirvana Meratnia},
  booktitle = {ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year = {2024},
  pages = {5805--5809},
  doi = {10.1109/ICASSP48485.2024.10447174},
  url = {https://arxiv.org/abs/2401.14211},
  eprint = {2401.14211},
  archivePrefix = {arXiv}
}

Keep exploring

FedLN

Learning together when labels cannot be trusted

FedCode

Exchange codebooks to make federated communication smaller