DeltaMask: Minimizing Communication in Federated Fine-Tuning of Vision Foundation Models

Vasileios Tsouvalas; Yuki M. Asano; Aaqib Saeed

DeltaMaskIEEE BigData 2024

DeltaMask: Minimizing Communication in Federated Fine-Tuning of Vision Foundation Models

Adapt foundation models by communicating sparse mask changes.

Vasileios Tsouvalas Yuki M. Asano Aaqib Saeed

Paper Code Poster

At a glance

Adapt a frozen model by communicating mask changes. DeltaMask trains stochastic masks over pretrained weights, selects changes to a shared reference mask, and encodes their positions with probabilistic filters. The server approximately reconstructs those masks and updates global mask probabilities.

One federated round · Mask updates

ServerGlobal mask

Probabilities + seed

Client ALocal mask

Client BClient C

SERVER → CLIENTS

One shared starting point

Frozen backboneShared seed

Position

0102030405060708

Global probabilities

80%70%20%60%30%80%20%80%

Shared-seed sampling ↓

Shared reference

11010101

Server = Clients · Same reference

Global probabilities · Shared seed · Shared reference

ServerGlobal mask

Local training

Client ALocal mask

Client BClient C

ON EACH CLIENT

Local mask learning

Local dataMask-score learningFrozen backbone

Position

0102030405060708

Before training

80%70%20%60%30%80%20%80%

After training

85%15%20%45%90%80%60%85%

Local mask sample

10001111

1 · Weight on 0 · Weight off

Local data · Learned probabilities · Frozen weights

ServerGlobal mask

Local selection

Client ALocal mask

Client BClient C

ON EACH CLIENT

Rank → select

Position

0102030405060708

Shared reference

11010101

Local mask

10001111

Changed positions · KL ranking

Position05Position02Position07Position04Omit

Top 3 of 4κ = 75%

Keep · 05, 02, 07 Omit · 04

Mask difference → KL ranking → Top-κ selection

ServerGlobal mask

PNG upload

Client ALocal mask

Client BClient C

CLIENTS → SERVER

A compact client upload

Selected changes

Position05Position02Position07

Position indices

Build BFuse88-bit fingerprints

Lossless compression

PNG

Compact upload

Upload · Encoded positions Weights · Stay local

Selected positions → BFuse8 → Lossless PNG

ServerGlobal mask

Reconstruction

Client ALocal mask

Client BClient C

ON THE SERVER

Reconstruct the client mask

PNG

Decompress PNGLossless decodingQuery BFuse8

Hit · 1 → flipMiss · 0 → keep

Position

0102030405060708

Shared reference

11010101

Filter says flip?

Reconstructed A

02 1 → 005 0 → 107 0 → 104 1 → 1 · omitted

Approximate recovery · Omitted changes · Rare false positives

ServerGlobal mask

Next probabilities

Client ALocal mask

Client BClient C

SERVER → NEXT ROUND

Aggregate → broadcast

Position

0102030405060708

Client A recovered

10011111

Client B recovered

11010111

Client C recovered

10011101

Bayesian aggregation ↓

Next probabilities

80%40%20%80%60%80%60%80%

Next round · New probabilities → new reference

Client masks + Beta(1,1) prior → Next probabilities

What this adds

01
Mask-based adaptation: learn subnetworks while keeping pretrained backbone weights fixed.
02
Sparse change encoding: rank differing positions and compress them with a binary fuse filter.
03
Controllable trade-off: vary retained changes and fingerprint width to balance bitrate and accuracy.

Abstract

Foundation Models (FMs) have revolutionized machine learning with their adaptability and high performance across tasks; yet, their integration into Federated Learning (FL) is challenging due to substantial communication overhead from their extensive parameterization. Current communication-efficient FL strategies, such as gradient compression, reduce bitrates to around 1 bit-per-parameter (bpp). However, these approaches fail to harness the characteristics of FMs, with their large number of parameters still posing a challenge to communication efficiency, even at these bitrate regimes. In this work, we present DeltaMask, a novel method that efficiently fine-tunes FMs in FL at an ultra-low bitrate, well below 1 bpp. DeltaMask employs stochastic masking to detect highly effective subnetworks within FMs and leverage stochasticity and sparsity in client masks to compress updates into a compact grayscale image using probabilistic filters, deviating from traditional weight training approaches. Our comprehensive evaluations across various datasets and architectures demonstrate DeltaMask efficiently achieves bitrates as low as 0.09 bpp, enhancing communication efficiency while maintaining FMs performance, as measured on 8 datasets and 5 pre-trained models of various network architectures.

DeltaMask

Optimize a stochastic mask

Transform unconstrained mask scores with a sigmoid, sample binary activations, and apply them to fixed pretrained weights. The classifier first receives one round of linear probing.

θ = sigmoid (s), m \sim Bern (θ), w^{'} = m ⊙ w_{init}

Sections III-A–C and Algorithm 1, PDF pages 4–6. The experiments target the last five backbone blocks.

Form and rank mask differences

Sample the global reference with a shared seed. Collect positions where local and reference bits disagree, rank them by relative entropy of their probabilities, and retain a top-κ subset.

Δ = {i : m_{local, i} \neq m_{ref, i}}

Section III-B and Equation 4, PDF page 5, with simplified round/client notation. The main setup schedules κ from 0.8.

Filter first, compress bytes second

Encode selected positions in a four-wise binary fuse filter using 8-bit fingerprints by default. PNG-like DEFLATE compression then shrinks the fingerprint bytes losslessly.

Section III-B, PDF page 5. Selection and probabilistic membership make mask recovery approximate; DEFLATE itself does not discard fingerprint data.

Recover masks and update probabilities

Query membership for every possible position, flip the corresponding reference bits, and feed reconstructed client masks to Bayesian aggregation for the next round.

\hat{m} = m_{ref} \oplus 𝟙_{\hat{Δ^{'}}}

Equation 5 and Algorithm 1, PDF pages 5–6. False positives can flip extra bits; this is not a lossless-recovery or formal privacy guarantee.

Method details in the paper ↗

Original paper figures & sources

DeltaMask pipeline: a client compares server and local binary masks, ranks changed positions, encodes them with a probabilistic filter and grayscale image, and the server reconstructs masks for Bayesian aggregation. — The update is a compact record of changed mask positions. Clients rank those changes and encode them with a probabilistic filter; the server queries the filter to reconstruct and aggregate local masks. Figure 2 in the accepted manuscript

Evaluation

Results

Accuracy at a fraction of the bitrate

Reported means · 8 image tasks · non-IID · 30 clients · 20% participation

Accuracy at a fraction of the bitrate. Reported mean accuracy (%) and Average bitrate (bits per parameter).
Method	Accuracy (%)	Bits / param.
FedMask	68.86	1.000
EDEN	78.56	0.715
DeepReduce	73.90	1.173
FedPM	81.64	0.948
DeltaMask	81.13	0.233

With full participation

Same 8 tasks · IID · CLIP ViT-B/32 · 30 clients · 100 rounds

IID data with full participation. CLIP ViT-B/32; eight image datasets, Dirichlet α=10, 30 clients, full participation, 100 rounds and one local epoch. Reported average accuracy (%) is higher-is-better; average bits per parameter (bpp) is lower-is-better.
Method	Average accuracy (%)	Average bitrate (bpp)
Fine-tuning	88.08	32
FedPM	86.07	0.872
DeltaMask	85.44	0.151

Reference

Cite this work

BibTeX

Download .bib

@inproceedings{tsouvalas2025deltamask,
  title = {DeltaMask: Minimizing Communication in Federated Fine-Tuning of Vision Foundation Models},
  author = {Vasileios Tsouvalas and Yuki M. Asano and Aaqib Saeed},
  booktitle = {2024 IEEE International Conference on Big Data (BigData)},
  year = {2025},
  pages = {8042--8051},
  publisher = {IEEE},
  doi = {10.1109/BigData62323.2024.10825398},
  url = {https://doi.org/10.1109/BigData62323.2024.10825398}
}

Keep exploring

AE-PSL

Align first. Compress every training step.

MPSL

Multimodal fine-tuning with lightweight edge clients.

DeltaMask: Minimizing Communication in Federated Fine-Tuning of Vision Foundation Models

One shared starting point

Local mask learning

Rank → select

A compact client upload

Reconstruct the client mask

Aggregate → broadcast

What this adds

Cite this work

Related publication

Keep exploring