SyncVL: Synchronizing Vision ⇔ Language Using Unsupervised Adaptation

Abstract

Contrastive vision–language models generalize well in zero-shot settings, but a persistent modality gap between their visual and textual embeddings limits how well that generalization transfers to specialized or shifted data distributions. Prior fixes either fine-tune on labeled target data (risking overfitting) or fine-tune on pseudo-labels in a one-directional way (risking erosion of pretrained knowledge and confirmation bias).

SyncVL instead keeps the original VLM encoders completely frozen and trains two small encoder–decoder synchronizers, Vision-to-Text (V2T) and Text-to-Vision (T2V), that project both modalities into a common latent subspace. Training is fully label-free: groups of image augmentations and text templates are formed, and a reward objective inspired by Group Relative Policy Optimization (GRPO) pushes each synchronizer toward outputs that are mutually consistent within a group and well-aligned across modalities, while a knowledge-distillation reward keeps the two synchronizers coherent with one another.

Across 24 benchmark datasets spanning unsupervised adaptation, out-of-distribution matching, unsupervised clustering, cross-modal retrieval, object detection, and segmentation, SyncVL improves consistently over state-of-the-art baselines and over six different VLM backbone, establishing a new reference point for label-free, encoder-frozen VLM adaptation.

Method

How vision and language stay in sync

SyncVL forms groups — k augmented views of one image, or m text templates of one class — and trains each synchronizer to be internally consistent across its group while aligning with the other modality's pseudo-class centers.

Overview of the bidirectional V2T and T2V synchronizers mapping vision and text embeddings into a common latent subspace — **Fig. 1 —** The V2T synchronizer maps visual embeddings toward textual pseudo-class centers under a consistency + alignment reward; the T2V synchronizer does the reverse, additionally guided by a knowledge-distillation reward from V2T.

V2T f_v2t

k augmentations of an image form a group. Each augmented embedding is mapped into the text-aligned space; the KL divergence between each instance's class distribution and the group average enforces consistency, while a contrastive term pulls it toward the majority (pseudo-positive) text class center and away from the runner-up (hard negative) class.

T2V f_t2v

m text templates of a class form a group and are mapped into the visual space. The same consistency + alignment rewards apply against visual pseudo-class centers derived from f_v2t, plus an L1 knowledge-distillation term that aligns T2V's latent encoder with V2T's — the two synchronizers are trained alternately across iterations until convergence.

Detailed SyncVL architecture diagram showing the V2T and T2V synchronizer pipelines with textual templates, augmentation groups, and reward functions — **Figure 2: —** Proposed bi-directional vision-language synchronization framework (SyncVL): Class names are expanded into text templates and encoded by the frozen text encoder g_φ; Unlabeled images are augmented and encoded by the frozen image encoder f_θ. The V2T synchronizer and T2V synchronizer are optimized with their respective consistency, alignment, and KD rewards, converging on a common sub-space used at inference.

Results

Consistent Gains across Datasets and Backbones

Unsupervised adaptation Results

Performance comparison with SOTA unsupervised adaptation methods.

Table 1: — Comparison of SOTA unsupervised adaptation methods using the CLIP ViT-B/32 backbone.

SyncVL added on top of six VLM backbone families

CLIP (B/16)

71.36

+ SyncVL

84.85

SigLIP (L/16)

56.38

+ SyncVL

68.71

ImageBind (H/14)

78.36

+ SyncVL

80.89

EVA-CLIP (B/16)

81.59

+ SyncVL

84.86

SigLIP-2 (L/16)

81.68

+ SyncVL

85.91

TULIP (B/16)

77.99

+ SyncVL

83.17

Average accuracy over CIFAR-10/100, DTD, UCF101, EuroSAT, Food101, Pets (Table 2). Gains hold even for the weakest starting point (SigLIP), where SyncVL adds +12.3 points on average.

Citation

BibTeX

@inproceedings{marrium_syncvl,
  title     = {SyncVL: Synchronizing Vision <=> Language Using Unsupervised Adaptation},
  author    = {Marrium, Maria and Khan, Muhammad Haris and Javed, Sajid and Mahmood, Arif},
  booktitle = {ECCV},
  year      = {2026}
}