Vision-Language Models

Vision-language models are the single most impactful concrete application of multimodal learning — jointly embedding images and text into a shared representation space, enabling exactly the zero-shot classification behavior previewed in Zero-Shot Learning.

CLIP's Training Objective — Contrastive Learning, Across Modalities

CLIP (Contrastive Language-Image Pretraining) trains on a large dataset of (image, caption) pairs, using an objective structurally almost identical to SimCLR's InfoNCE loss — except instead of contrasting two augmented views of the same image, CLIP contrasts an image against its matching caption text.

\(\mathbf{i}_j\) is image \(j\)'s embedding (from an image encoder, typically a CNN or Vision Transformer), \(\mathbf{t}_j\) is its matching caption's embedding (from a text encoder, structurally similar to Transformer Encoder). Within a batch, the true (image, caption) pairs are the positives; every mismatched image-caption combination in the batch serves as a negative — the model is trained so matching pairs' embeddings end up close together, and mismatched pairs' embeddings end up far apart.

Two Separate Encoders, One Shared Space

Code — Computing CLIP Similarity Directly

import torch
from transformers import CLIPModel, CLIPProcessor

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

inputs = processor(
    text=["a photo of a mountain", "a photo of a beach"],
    images=some_image, return_tensors="pt", padding=True
)
outputs = model(**inputs)
image_embeds = outputs.image_embeds   # from the image encoder
text_embeds = outputs.text_embeds       # from the text encoder -- SAME shared dimensionality
similarity = image_embeds @ text_embeds.T   # a dot product in the shared space

Why This Enables So Much Downstream Utility

Once images and text share one consistent embedding space, an enormous range of tasks become possible without any task-specific training: zero-shot classification (comparing an image to candidate label descriptions), image search via text query, and — critically — serving as the text-conditioning backbone in text-to-image diffusion models (recall Diffusion Conditioning's text encoder), among many other applications built on top of this shared representation.

Common Mistakes

• Assuming CLIP's image and text encoders are identical or share weights — they're two entirely separate networks, specialized to their own modality, trained jointly only through the shared contrastive loss, not through weight sharing.
• Expecting CLIP to generate images or text — it's fundamentally a representation/alignment model, not a generative one; its embeddings are commonly used by separate generative models (like Stable Diffusion) rather than generating content itself.

Interview Relevance

Q: "How does CLIP's training objective relate to SimCLR's contrastive learning approach?" Both use essentially the same InfoNCE-style contrastive loss, pulling matching pairs together and pushing mismatched pairs apart within a batch. SimCLR contrasts two augmented views of the same image against other images; CLIP contrasts an image against its matching caption text, treating every mismatched image-caption combination in the batch as a negative — the same underlying contrastive mechanism, applied across two different modalities instead of within one.

Practice Question

Why does CLIP need two separate encoders rather than one shared network processing both images and text together?

Related DL Notes

• SimCLR
• Zero-Shot Learning
• Diffusion Conditioning