Specialized image domains can benefit from adaptation before labels are available. SDAPT adapts a shared vision backbone from local, unlabeled images, using frozen client layers and server-side self-supervision. It then freezes the adapted backbone and returns representations for each client to train its own supervised task head.
Adapt together · Specialize locally
ClientsImages + labels stay local
Phase 1 of 2Domain adaptation
Shared serverStudent + EMA teacher
Unlabeled domain adaptation
Local images → shared feature batch
AClient AUnlabeled images
Local image
2 global
8 local
Client partitionPatch embed + first block
Frozen
CLS + patch tokens
BClient BUnlabeled images
Local image
2 global
8 local
Client partitionPatch embed + first block
Frozen
CLS + patch tokens
One server batchA + B + …
Two global + eight local crops · Frozen client layers · Features to the server
ClientsImages + labels stay local
Phase 1 of 2Domain adaptation
Shared serverStudent + EMA teacher
Unlabeled domain adaptation
Self-distill on the shared server
All client feature batchesShared centering statistics
Global viewsTeacherEMA weights
EMA
Global + local viewsStudentLoRA + last block
Mask global patches
Teacher targets ↓Student outputs ↓
Self-supervised lossDINO + λ₁ iBOT + λ₂ KoLeo
Update studentServer only
DINO + iBOT + KoLeo · Server gradients · EMA teacher update
Original image → frozen backbone → adapted representation → client
ClientsImages + labels stay local
Phase 2 of 2Task-head training
Shared serverAdapted teacher
Labeled downstream tasks
One adapted backbone, local task heads
AClient ATask A
Returned representation
Task headTrainable
Prediction+Local labelSupervised loss
Update this head
BClient BTask B
Returned representation
Task headTrainable
Prediction+Local labelSupervised loss
Update this head
Backbone stays frozenIndependent local heads · Parallel training
Returned features + local labels · Train each head independently
1 / 4
What this adds
01
Adapt without client backpropagation: frozen client layers send features to a shared DINOv2 student and EMA teacher.
02
Separate adaptation from task learning: one adapted backbone serves local heads with different in-domain tasks.
03
Expose the budget trade-off: compare multi-crop self-distillation with distributed alternatives across three image domains.
Abstract
Vision Foundation Models (VFMs) support tasks with limited labeled data, although their performance can decline in specialized domains. Domain-Adaptive Pretraining (DAPT) can address this limitation by adapting their representations using unlabeled in-domain images; yet these often are privacy-sensitive and distributed across resource-constrained devices. Split Learning (SL) addresses these constraints by offloading computation to a server while keeping raw images local. Existing SL methods, however, explore self-supervised training from scratch, leaving collaborative DAPT of VFMs underexplored. For this, we propose SDAPT, which performs server-side DAPT with LoRA adapters using features from frozen client layers, then trains task-specific heads on the frozen adapted VFM, keeping images and labels local. Across 3 domains, and diverse few-shot settings, we observe mean gains of 7.4 pp over direct fine-tuning, while adapting once on unlabeled domain data yields shared representations that improve performance across multiple downstream tasks.
Each client creates two global and eight local crops of an unlabeled in-domain image. The frozen client partition produces a CLS activation and patch tokens for every view. Only these activations go to the server; the student and teacher share the client partition.
Section 2.1, Domain Adaptive Pretraining Phase, and Figure 1 · p. 2. Section 3 · p. 3 uses DINOv2 ViT-B/14 with the first transformer block on the client.
02
Adapt one student on the combined batch
The server concatenates client features into a global batch for DINOv2 self-supervision and batch-level centering. The teacher consumes global views; the student consumes global and local views, with iBOT masking applied to global patch tokens on the server. Gradients update the student, and an exponential moving average updates the teacher.
Equation (1), Figure 1 and Section 2.1 · pp. 2–3. The server uses LoRA and fully unfreezes its last block; the CheXpert experiments unfreeze the last two. Frozen client layers require no returned gradients.
03
Freeze the adapted teacher for downstream features
After adaptation, the full split backbone is frozen. Clients pass original, non-augmented images through their frozen partition. The server uses the adapted teacher to return CLS features for classification, or patch features for dense tasks.
Section 2.1, Fine-tuning Phase, and Figure 1 · pp. 2–3. Dense prediction is described by the method; the reported experiments cover classification only.
04
Train task-specific heads locally
Each client combines the returned representation with its own labels to train a supervised task head. Only the head changes, so clients can train in parallel without sending loss gradients to the server.
Equation (2) and Section 2.1 · p. 3. rn(x) is the returned representation; gϕn is client n’s head and y its local label.
@misc{vanhonk2026sdapt,
title = {Domain Adaptive Pretraining in Split Learning for Vision Foundation Models},
author = {Stan van Honk and Vasileios Tsouvalas and Nirvana Meratnia},
year = {2026},
note = {Manuscript},
url = {https://nousphera.github.io/projects/sdapt/}
}