Multimodal fine-tuning with lightweight edge clients.
Multimodal fine-tuning can overwhelm an edge device even when few parameters change. MPSL keeps lightweight tokenizers on clients and runs a shared transformer on the server, coordinating training through local losses and returned gradients.
Multimodal splitting: support paired image, audio and text inputs with one shared encoder.
Shared optimization: aggregate client losses for one server backward pass while keeping raw labels local.
Task-dependent evidence: measure accuracy, retrieval, client compute and communication across seven datasets.
Multimodal transformers integrate diverse data types like images, audio, and text, advancing tasks such as audio-visual understanding and image-text retrieval; yet their high parameterization limits deployment on resource-constrained edge devices. Split Learning (SL), which partitions models at a designated cut-layer to offload compute-intensive operations to the server, offers a promising approach for distributed training of multimodal transformers, though its application remains underexplored. We present MPSL, a parallel SL approach for computational efficient fine-tuning of multimodal transformers in a distributed manner, while eliminating label sharing, client synchronization, and per-client sub-model management. MPSL employs lightweight client-side tokenizers and a unified modality-agnostic encoder, allowing flexible adaptation to task-specific needs. Our evaluation across 7 multimodal datasets demonstrates that MPSL matches or outperforms Federated Learning, reduces client-side computations by 250×, and achieves superior scalability in communication cost with model growth. Through extensive analysis, we highlight task suitability, trade-offs, and scenarios where MPSL excels, inspiring further exploration.
Read more ↗Image patches, audio spectrogram patches and text tokens provide compatible inputs to a modality-agnostic transformer. Clients run the tokenizers; the server runs the encoder and task head.
Section 3.1–3.2 and Figure 1 · public preprint, pp. 2–4.
Early fusion concatenates token sequences before encoding. Late fusion encodes modalities independently with shared weights and concatenates their representations afterward. The useful choice depends on the task and modality pair.
Section 3.2 and Table 5 · pp. 3 and 7.
Predictions return to clients for loss calculation. The server weights each loss by its share of the global batch, backpropagates once, then returns cut-layer gradients to the local tokenizers.
Equation (4), p. 4. ℬn is client n’s batch; ℬ is the combined batch. This avoids client-weight synchronization during training, not all coordination.
After training, the reported comparisons average the client tokenizers and join them with the shared server model. This post-training aggregation is distinct from the loss-based training loop.
Section 3.3 · p. 4.
Mean accuracy · 5 classification tasks · 100 clients
| Method | Accuracy (%) | Params (M) |
|---|---|---|
| FedAvg | 67.20 | 43.7 |
| FedCLIP | 45.74 | 3.1 |
| MPSL | 66.64 | 1.0 |
100 clients · accuracy (%) · 3-seed means · GFLOPs per input
| Method | COCO-QA (%) | MELD (%) | Client GFLOPs |
|---|---|---|---|
| Centralized | 61.7 ± 0.1 | 61.4 ± 0.4 | 25.8 |
| FedAvg | 53.5 ± 0.2 | 48.9 ± 0.0 | 25.8 |
| FedCLIP | 13.1 ± 2.6 | 50.7 ± 0.7 | 25.8 |
| MPSL | 46.2 ± 0.5 | 58.2 ± 0.9 | 0.1 |
@inproceedings{fudala2025mpsl,
title = {Fine-Tuning Multimodal Transformers on Edge: A Parallel Split Learning Approach},
author = {Timo Fudala and Vasileios Tsouvalas and Nirvana Meratnia},
booktitle = {FedGenAI Workshop at IJCAI 2025},
year = {2025},
url = {https://arxiv.org/abs/2502.06355},
eprint = {2502.06355},
archivePrefix = {arXiv}
}