Adapt foundation models by communicating sparse mask changes.
Adapt a frozen model by communicating mask changes. DeltaMask trains stochastic masks over pretrained weights, selects changes to a shared reference mask, and encodes their positions with probabilistic filters. The server approximately reconstructs those masks and updates global mask probabilities.
Mask-based adaptation: learn subnetworks while keeping pretrained backbone weights fixed.
Sparse change encoding: rank differing positions and compress them with a binary fuse filter.
Controllable trade-off: vary retained changes and fingerprint width to balance bitrate and accuracy.
Foundation Models (FMs) have revolutionized machine learning with their adaptability and high performance across tasks; yet, their integration into Federated Learning (FL) is challenging due to substantial communication overhead from their extensive parameterization. Current communication-efficient FL strategies, such as gradient compression, reduce bitrates to around 1 bit-per-parameter (bpp). However, these approaches fail to harness the characteristics of FMs, with their large number of parameters still posing a challenge to communication efficiency, even at these bitrate regimes. In this work, we present DeltaMask, a novel method that efficiently fine-tunes FMs in FL at an ultra-low bitrate, well below 1 bpp. DeltaMask employs stochastic masking to detect highly effective subnetworks within FMs and leverage stochasticity and sparsity in client masks to compress updates into a compact grayscale image using probabilistic filters, deviating from traditional weight training approaches. Our comprehensive evaluations across various datasets and architectures demonstrate DeltaMask efficiently achieves bitrates as low as 0.09 bpp, enhancing communication efficiency while maintaining FMs performance, as measured on 8 datasets and 5 pre-trained models of various network architectures.
Read more ↗Transform unconstrained mask scores with a sigmoid, sample binary activations, and apply them to fixed pretrained weights. The classifier first receives one round of linear probing.
Sections III-A–C and Algorithm 1, PDF pages 4–6. The experiments target the last five backbone blocks.
Sample the global reference with a shared seed. Collect positions where local and reference bits disagree, rank them by relative entropy of their probabilities, and retain a top-κ subset.
Section III-B and Equation 4, PDF page 5, with simplified round/client notation. The main setup schedules κ from 0.8.
Encode selected positions in a four-wise binary fuse filter using 8-bit fingerprints by default. PNG-like DEFLATE compression then shrinks the fingerprint bytes losslessly.
Section III-B, PDF page 5. Selection and probabilistic membership make mask recovery approximate; DEFLATE itself does not discard fingerprint data.
Query membership for every possible position, flip the corresponding reference bits, and feed reconstructed client masks to Bayesian aggregation for the next round.
Equation 5 and Algorithm 1, PDF pages 5–6. False positives can flip extra bits; this is not a lossless-recovery or formal privacy guarantee.
Results from the source manuscript ↗
DeltaMask reports 85.44% average accuracy at 0.151 bpp: 0.63 percentage points below FedPM and 2.64 below the fine-tuning reference.
CLIP ViT-B/32; eight image datasets, Dirichlet α=10, 30 clients, full participation, 100 rounds and one local epoch. Reported average accuracy (%) is higher-is-better; average bits per parameter (bpp) is lower-is-better.
The table reports no uncertainty for the eight-task averages. The fine-tuning reference updates the same last five blocks targeted by masking, not the entire backbone.
| Method | Average accuracy (%) | Average bitrate (bpp) |
|---|---|---|
| Fine-tuning | 88.08 | 32 |
| FedMask | 79.23 | 1.0 |
| EDEN | 84.14 | 0.691 |
| DeepReduce | 76.53 | 1.089 |
| FedPM | 86.07 | 0.872 |
| DeltaMask | 85.44 | 0.151 |
With 20% participation, DeltaMask reports 81.13% at 0.233 bpp: 0.51 percentage points below FedPM and 2.06 below fine-tuning.
CLIP ViT-B/32; eight image datasets, Dirichlet α=0.1, 30 clients with 20% participation, 300 rounds and one local epoch. Average accuracy (%) is higher-is-better; average bpp is lower-is-better.
These averages have no tabulated uncertainty. This setting uses three times as many rounds as the IID panel, so bitrate alone is not total training communication.
| Method | Average accuracy (%) | Average bitrate (bpp) |
|---|---|---|
| Fine-tuning | 83.19 | 32 |
| FedMask | 68.86 | 1.0 |
| EDEN | 78.56 | 0.715 |
| DeepReduce | 73.90 | 1.173 |
| FedPM | 81.64 | 0.948 |
| DeltaMask | 81.13 | 0.233 |
All five backbones use approximately 0.2–0.25 bpp. Only CLIP ViT-L/14 exceeds its fine-tuning accuracy in this separate ten-client study.
CIFAR-100, IID Dirichlet α=10, ten clients. Accuracy (%) is higher-is-better; DeltaMask bitrate is lower-is-better. Entries retain the source’s ± terms; experiments average three runs, but the dispersion convention is not defined.
This is a separate ten-client experiment; its CLIP ViT-B/32 values should not be substituted into the thirty-client tables. Unusually small ± terms are reproduced without reinterpretation.
| Backbone | Fine-tuning accuracy (%) | DeltaMask accuracy (%) | DeltaMask bitrate (bpp) |
|---|---|---|---|
| CLIP ViT-B/32 | 77.35 ± 0.009 | 75.82 ± 0.023 | 0.207 ± 0.001 |
| CLIP ViT-L/14 | 89.07 ± 0.012 | 89.48 ± 0.031 | 0.225 ± 0.002 |
| DINOv2-Base | 75.01 ± 0.007 | 73.36 ± 0.027 | 0.197 ± 0.001 |
| DINOv2-Small | 65.55 ± 0.019 | 63.01 ± 0.033 | 0.214 ± 0.001 |
| ConvMixer-768/32 | 78.52 ± 0.009 | 75.31 ± 0.021 | 0.251 ± 0.001 |
@inproceedings{tsouvalas2025deltamask,
title = {DeltaMask: Minimizing Communication in Federated Fine-Tuning of Vision Foundation Models},
author = {Vasileios Tsouvalas and Yuki M. Asano and Aaqib Saeed},
booktitle = {2024 IEEE International Conference on Big Data (BigData)},
year = {2025},
pages = {8042--8051},
publisher = {IEEE},
doi = {10.1109/BigData62323.2024.10825398},
url = {https://doi.org/10.1109/BigData62323.2024.10825398}
}