Multimodal fine-tuning with lightweight edge clients.
Multimodal fine-tuning can overwhelm an edge device even when few parameters change. MPSL keeps lightweight tokenizers on clients and runs a shared transformer on the server, coordinating training through local losses and returned gradients.
Multimodal splitting: support paired image, audio and text inputs with one shared encoder.
Shared optimization: aggregate client losses for one server backward pass while keeping raw labels local.
Task-dependent evidence: measure accuracy, retrieval, client compute and communication across seven datasets.
Multimodal transformers integrate diverse data types like images, audio, and text, advancing tasks such as audio-visual understanding and image-text retrieval; yet their high parameterization limits deployment on resource-constrained edge devices. Split Learning (SL), which partitions models at a designated cut-layer to offload compute-intensive operations to the server, offers a promising approach for distributed training of multimodal transformers, though its application remains underexplored. We present MPSL, a parallel SL approach for computational efficient fine-tuning of multimodal transformers in a distributed manner, while eliminating label sharing, client synchronization, and per-client sub-model management. MPSL employs lightweight client-side tokenizers and a unified modality-agnostic encoder, allowing flexible adaptation to task-specific needs. Our evaluation across 7 multimodal datasets demonstrates that MPSL matches or outperforms Federated Learning, reduces client-side computations by 250×, and achieves superior scalability in communication cost with model growth. Through extensive analysis, we highlight task suitability, trade-offs, and scenarios where MPSL excels, inspiring further exploration.
Read more ↗Image patches, audio spectrogram patches and text tokens provide compatible inputs to a modality-agnostic transformer. Clients run the tokenizers; the server runs the encoder and task head.
Section 3.1–3.2 and Figure 1 · public preprint, pp. 2–4.
Early fusion concatenates token sequences before encoding. Late fusion encodes modalities independently with shared weights and concatenates their representations afterward. The useful choice depends on the task and modality pair.
Section 3.2 and Table 5 · pp. 3 and 7.
Predictions return to clients for loss calculation. The server weights each loss by its share of the global batch, backpropagates once, then returns cut-layer gradients to the local tokenizers.
Equation (4), p. 4. ℬn is client n’s batch; ℬ is the combined batch. This avoids client-weight synchronization during training, not all coordination.
After training, the reported comparisons average the client tokenizers and join them with the shared server model. This post-training aggregation is distinct from the loss-based training loop.
Section 3.3 · p. 4.
Results from arXiv:2502.06355v3 ↗
MPSL sharply lowers client compute and improves MELD over FedAvg, but trails FedAvg by 7.3 points on COCO-QA and 4.4 on Kinetics-Sounds.
Meta-Transformer ViT-B/16; 100 clients with Dirichlet α=0.1. Selected classification tasks, accuracy (%) with reported ± terms over three seeds; higher is better. Client GFLOPs per input are lower-is-better.
These tasks expose both gains and deficits; they are three of five classification datasets. GFLOPs measure client work, not total system work or latency. The paper does not specify the type of ± interval.
| Method | COCO-QA (%) | Kinetics-Sounds (%) | MELD (%) | Client GFLOPs |
|---|---|---|---|---|
| Centralized | 61.7 ± 0.1 | 81.8 ± 0.2 | 61.4 ± 0.4 | 25.8 |
| FedAvg | 53.5 ± 0.2 | 71.4 ± 0.4 | 48.9 ± 0.0 | 25.8 |
| FedCLIP | 13.1 ± 2.6 | 63.5 ± 0.8 | 50.7 ± 0.7 | 25.8 |
| MPSL | 46.2 ± 0.5 | 67.0 ± 1.1 | 58.2 ± 0.9 | 0.1 |
MPSL improves on both evaluated distributed alternatives but remains well below centralized recall on both datasets.
Image-to-text retrieval, 100 clients; top-1 recall (%) with reported ± terms over three seeds, higher is better. Full ViT-B/16 encoder fine-tuning; raw recall, not normalized scores.
The table also reports text-to-image retrieval and R@10; the selected rows isolate one direction and recall threshold. The reported ± convention is unspecified.
| Method | MS-COCO R@1 (%) | Flickr30K R@1 (%) |
|---|---|---|
| Centralized | 56.8 ± 2.4 | 66.8 ± 5.3 |
| FedAvg | 1.5 ± 0.1 | 11.2 ± 1.7 |
| FedCLIP | 0.1 ± 0.0 | 0.1 ± 0.0 |
| MPSL | 21.1 ± 0.8 | 26.0 ± 0.5 |
Larger batches substantially improve normalized recall over batch size 50, although the Flickr30K trend is not monotonic.
MPSL with 25 clients. Top-1 retrieval recall normalized to centralized fine-tuning (% of reference performance); higher is better. These values are ratios, not raw recall.
No uncertainty is tabulated. Larger retrieval batches are marked infeasible because of dataset partition sizes; this ablation uses fewer clients than the raw-recall table.
| Batch size | Flickr30K normalized recall (%) | MS-COCO normalized recall (%) |
|---|---|---|
| 50 | 37.9 | 21.3 |
| 100 | 69.5 | 53.2 |
| 150 | 63.0 | 64.1 |
| 200 | 69.6 | 69.7 |
@inproceedings{fudala2025mpsl,
title = {Fine-Tuning Multimodal Transformers on Edge: A Parallel Split Learning Approach},
author = {Timo Fudala and Vasileios Tsouvalas and Nirvana Meratnia},
booktitle = {FedGenAI Workshop at IJCAI 2025},
year = {2025},
url = {https://arxiv.org/abs/2502.06355},
eprint = {2502.06355},
archivePrefix = {arXiv}
}