Many-Task Federated Fine-Tuning via Unified Task Vectors

Vasileios Tsouvalas; Tanir Ozcelebi; Nirvana Meratnia

MaTUFedGenAI @ IJCAI · 2025

Many-Task Federated Fine-Tuning via Unified Task Vectors

Many local tasks. One shared task vector.

Vasileios Tsouvalas Tanir Ozcelebi Nirvana Meratnia

Paper Code Poster

At a glance

Clients with several tasks produce updates that can interfere. MaTU sends one unified task vector plus per-task masks and scalars, preserving task differences while enabling server aggregation within tasks and transfer between related tasks.

Inside MaTU

Measure each task from a common originClients train their assigned tasks individually. Each task vector is the difference between its fine-tuned weights and the same pretrained model, so all task updates share a reference point.

Elect a sign and aligned magnitudeFor every coordinate, MaTU takes the sign of the sum of task updates. It then keeps the largest magnitude among entries with that sign. The schematic below illustrates one coordinate.

Attach a mask and scalar for each taskA binary mask keeps coordinates whose signs agree with a task’s update; a scalar corrects the total magnitude. Clients send one unified vector together with these compact task-specific modulators.

Aggregate within and across related tasksThe server first combines clients that share a task, using agreement masks. It then transfers updates from up to κ other tasks whose sign similarity exceeds 0.5, applying the receiving task’s mask.

Recover task-specific updates for the next roundThe server reunifies updated task vectors for each client. On return, each task multiplies the unified vector by its mask and scalar, then adds the recovered update to pretrained weights. Recovery is approximate.

What this adds

01
Compact task representation: elect a shared vector and retain lightweight task-specific modulators.
02
Selective transfer: combine same-task updates with sign-aligned related tasks.
03
No retained client weights: the server computes task-level updates without storing client-specific models across rounds.

Abstract

Federated Learning (FL) traditionally assumes homogeneous client tasks; however, in real-world scenarios, clients often specialize in diverse tasks, introducing task heterogeneity. To address this challenge, Many-Task FL (MaT-FL) has emerged, enabling clients to collaborate effectively despite task diversity. Existing MaT-FL approaches rely on client grouping or personalized layers, requiring the server to manage individual models and failing to account for clients handling multiple tasks. We propose MaTU, a MaT-FL approach that enables joint learning of task vectors across clients, eliminating the need for clustering or client-specific weight storage at the server. MaTU constructs a single “unified” task vector per client and leverages lightweight task-specific modulators — binary masks and scalar rescalers — to disentangle and adapt per-task behavior. This design ensures stateless server aggregation and naturally scales to large task sets with minimal communication overhead. Evaluated across 30 datasets, MaTU achieves superior performance over state-of-the-art MaT-FL approaches, with results comparable to per-task fine-tuning, while delivering significant communication savings.

MaTU

Train from a shared reference

Each client fine-tunes its assigned tasks separately. A task vector is the change from the common pretrained weights. The experiments use ViT-B/32 with rank-16 LoRA on vision classification tasks.

τ_{t} = θ_{t}^{*} - θ_{p}

Section 3.1 · p. 2. θ_t^* denotes fine-tuned task weights and θ_p the shared pretrained origin.

Elect the unified vector

At each coordinate, take the sign of the sum of local task vectors. Among entries agreeing with that sign, retain the largest magnitude. This election preserves a common direction without averaging away its amplitude.

τ = σ ⊙ μ

Equation (2) · p. 3. σ is the elected sign, μ the largest aligned magnitude; ⊙ denotes elementwise multiplication.

Retain task-specific modulation

Each task keeps a binary mask for entries aligned with its own signs and a scalar matching its original total absolute magnitude. These modulators produce approximate task updates from the unified vector.

{\dot{τ}}_{t} = λ_{t} m_{t} ⊙ τ, θ_{t} = θ_{p} + {\dot{τ}}_{t}

Section 3.2, “Forming clients updates” · p. 4. Client and round indices are omitted consistently; m_t is a task mask and λ_t its scalar.

Aggregate and return updates

The server uses task agreement masks to combine same-task clients, then transfers sign-aligned updates from up to κ related tasks with similarity above 0.5. It reunifies the updated vectors for each client and returns new masks and scalars for the next local-training round.

Equations (3)–(7) · p. 4. “Stateless” means no client-specific weights retained across rounds; task-level aggregation still occurs. Cross-task transfer is not uniformly beneficial in the ablations.

Method details in the paper ↗

Original paper figures & sources

MaTU pipeline showing a client combining three task vectors into one unified vector and task modulators, then a server aggregating masks and estimating task similarity before producing updated vectors. — Clients communicate one unified vector plus a binary mask and scalar for each task. The server combines same-task updates and transfers information between tasks whose update directions agree. Figure 1 in the public preprint

Evaluation

Results

Results from arXiv:2502.06376v3 ↗

One task per client

MaTU reaches 84.32% average accuracy, above the evaluated federated baselines and 5.89 percentage points below individual centralized fine-tuning.

Accuracy (%)

Higher is better

0100

Individual (centralized)

90.21

FedAvg

64.60

NTK-FedAvg

65.13

FedProx

67.14

FedPer

82.11

MaT-FL

76.66

MaTU

84.32

Traffic (Mbits / task / round)

Lower is better

Individual (centralized)

—

FedAvg

6.32

NTK-FedAvg

6.32

FedProx

6.32

FedPer

5.36

MaT-FL

6.32

MaTU

6.32

Experiment details

Eight vision tasks, ImageNet-21k-pretrained ViT-B/32 with rank-16 LoRA, 30 clients, 20% participation, one local epoch. Mean task accuracy (%), higher is better; reported bitrate in millions of bits per task per round, lower is better.

MaT-FL uses 300 rounds; the other federated methods use 100. The table supplies no uncertainty estimates, and per-round bitrate does not capture that round-count difference.

One task per client. Eight vision tasks, ImageNet-21k-pretrained ViT-B/32 with rank-16 LoRA, 30 clients, 20% participation, one local epoch. Mean task accuracy (%), higher is better; reported bitrate in millions of bits per task per round, lower is better.
Method	Average accuracy (%)	Bitrate (M bits/task/round)
Individual (centralized)	90.21	—
FedAvg	64.60	6.32
NTK-FedAvg	65.13	6.32
FedProx	67.14	6.32
FedPer	82.11	5.36
MaT-FL	76.66	6.32
MaTU	84.32	6.32

Table 1 · PDF page 6 ↗

Several tasks per client

MaTU exceeds MaT-FL by 11.87 percentage points while reporting 8.04 versus 20.43 million bits per task per round. It remains 10.74 points below individual centralized fine-tuning.

Accuracy (%)

Higher is better

0100

Individual (centralized)

90.21

FedAvg

51.63

NTK-FedAvg

56.06

FedProx

56.32

FedPer

48.37

MaT-FL

67.60

MaTU

79.47

Traffic (Mbits / task / round)

Lower is better

021

Individual (centralized)

—

FedAvg

20.43

NTK-FedAvg

20.43

FedProx

20.43

FedPer

17.16

MaT-FL

20.43

MaTU

8.04

Experiment details

Same eight-task ViT-B/32 benchmark; 30 clients, task concentration ζt=0.5, 20% participation and one local epoch. Accuracy (%) is averaged across tasks; bitrate is millions of bits per task per round.

MaT-FL again uses 300 rounds versus 100 for the other methods. No seed count or uncertainty is specified for these table entries.

Several tasks per client. Same eight-task ViT-B/32 benchmark; 30 clients, task concentration ζt=0.5, 20% participation and one local epoch. Accuracy (%) is averaged across tasks; bitrate is millions of bits per task per round.
Method	Average accuracy (%)	Bitrate (M bits/task/round)
Individual (centralized)	90.21	—
FedAvg	51.63	20.43
NTK-FedAvg	56.06	20.43
FedProx	56.32	20.43
FedPer	48.37	17.16
MaT-FL	67.60	20.43
MaTU	79.47	8.04

Table 2 · PDF page 6 ↗

Study scope & sources

Vision classification with one backbone and adapter configuration.
MaT-FL uses 300 rounds versus 100 for MaTU in the eight-task tables; no repeated-run uncertainty is reported.
Per-task, per-round bitrate is not total training traffic; the 30-task study reports normalized accuracy.

Data: An eight-task vision benchmark and a larger 30-task vision benchmark; both class and task heterogeneity are simulated.
Model: ImageNet-21k-pretrained ViT-B/32 with LoRA rank 16; task vectors share the same pretrained origin.
Compared with: Individual centralized fine-tuning, FedAvg, NTK-FedAvg, FedProx, FedPer and MaT-FL.
Additional analysis: Task-relatedness metrics, 2–30 tasks per client, deliberately conflicting task groups and cross-task aggregation ablations.

Scope & limitations

All empirical tasks are vision classification with one backbone and adapter configuration; language or heterogeneous model architectures are not evaluated.
The main tables report no uncertainty, and MaT-FL is allocated three times as many rounds as the other methods.
The 30-task figure reports accuracy normalized to individual task fine-tuning, unlike the raw accuracy in the eight-task tables.
Cross-task transfer is not uniformly beneficial: the no-cross-task variant slightly exceeds MaTU in the two-conflict ablation in Figure 6b.

arXiv:2502.06376v3, 8 July 2025; 9 pages; no appendix ↗

Reference

Cite this work

BibTeX

Download .bib

@inproceedings{tsouvalas2025matu,
  title = {Many-Task Federated Fine-Tuning via Unified Task Vectors},
  author = {Vasileios Tsouvalas and Tanir Ozcelebi and Nirvana Meratnia},
  booktitle = {FedGenAI Workshop at IJCAI 2025},
  year = {2025},
  url = {https://arxiv.org/abs/2502.06376},
  eprint = {2502.06376},
  archivePrefix = {arXiv}
}

Keep exploring

FedLN

Learning together when labels cannot be trusted

FiBA

Understanding invisible backdoors in collaborative learning