Self-supervised learning generates its own training labels directly from unlabeled data — by hiding part of the input and training the model to predict it — instead of relying on any human-provided labels.
The Core Trick: Manufacture a Label
| Domain | What's Hidden | What the Model Predicts |
|---|---|---|
| Text (language models) | A word or the next token in a sentence | The missing/next word — this is how GPT-style models pretrain |
| Images | A patch of the image, or its rotation | The missing patch, or the rotation applied |
| Video | Future frames | What happens next |
# Conceptual example: masked-word prediction (simplified)
sentence = "Machine learning models learn from ___"
target = "data" # the label came from the sentence itself — no human tagged itNo person had to sit down and write "this sentence is about machine learning" — the label ("data") was already present in the raw, unlabeled text. This is what makes self-supervised learning scale to billions of examples scraped from the web.
Why It Matters
Self-supervised pretraining is the technique behind most modern large language models and many vision models: the model first learns broad, general-purpose representations from massive unlabeled data, and is then fine-tuned on a smaller labeled dataset for a specific task. See Generative AI notes for how this connects to LLMs.
Self-Supervised vs Semi-Supervised — The Distinction That Confuses People
| Self-Supervised | Semi-Supervised | |
|---|---|---|
| Labels | Generated automatically from the data's own structure | A small set of real, human-provided labels |
| Human labeling effort | Zero (for the pretraining stage) | Some, for the labeled subset |
| Typical use | Pretraining a general-purpose model | Training a task-specific model with limited labels |
Practical Use Cases
- Pretraining large language models on raw text
- Pretraining vision models on unlabeled images before fine-tuning for a specific classification task
Limitations
- Requires very large amounts of data and compute to pretrain effectively
- The "pretext task" (e.g. predicting a masked word) must be designed so that solving it actually forces the model to learn useful, general representations
Common Mistakes
- Confusing self-supervised learning with unsupervised learning generally — self-supervised specifically manufactures a supervised-style label from the input; plain unsupervised learning (like clustering) never predicts any label at all.
Interview Relevance
Q: "How does self-supervised learning differ from unsupervised learning?" Self-supervised learning creates its own pseudo-labels from the data and trains with a supervised-style loss; unsupervised learning (e.g. K-Means) never uses labels of any kind.
Practice Question
Describe a self-supervised pretext task for a dataset of unlabeled product images that could help a model later learn to classify product categories with only a few hundred labeled examples.
