Data Augmentation

Data augmentation regularizes a model by expanding its effective training set — applying random, label-preserving transformations to existing examples so the model sees more variation without needing any newly collected data.

The Core Idea

Instead of training on a fixed image exactly as-is, apply a random transformation (a flip, a crop, a rotation, a brightness change) each time it's used in training — the model then sees a slightly different version of that same underlying example on every epoch, discouraging it from memorizing pixel-exact details that won't generalize and encouraging it to learn features robust to those variations instead.

Common Augmentations by Data Type

The Critical Constraint: Label Preservation

Every augmentation must preserve the example's true label — this sounds obvious, but it's a genuinely common source of subtle bugs. Flipping an image of a handwritten digit "6" horizontally can make it visually resemble "9" — a label-breaking augmentation that would actively corrupt training. Choosing which augmentations are valid always requires domain-specific judgment about what transformations genuinely leave the label unchanged.

Code

import torchvision.transforms as T

train_transforms = T.Compose([
    T.RandomHorizontalFlip(p=0.5),
    T.RandomRotation(degrees=15),
    T.ColorJitter(brightness=0.2, contrast=0.2),
    T.RandomCrop(size=224, padding=4),
    T.ToTensor(),
])

# Applied automatically each time an image is loaded during training
# validation/test transforms should NOT include random augmentations --
# only deterministic resizing/normalization, for consistent evaluation
val_transforms = T.Compose([
    T.Resize((224, 224)),
    T.ToTensor(),
])

Why Augmentation Only Applies to Training Data

Validation and test evaluation need to be deterministic and consistent to give a reliable, repeatable performance estimate — random augmentation would make the same example produce different predictions on different evaluation runs, undermining that reliability. Augmentation is specifically a training-time technique, applied only to the training set, mirroring the training-vs-evaluation-mode distinction already established for Dropout and BatchNorm.

Common Mistakes

• Applying an augmentation that breaks the label's validity for the specific task (e.g. horizontal flips on text-containing images, where flipping makes any text unreadable and often changes meaning) — always sanity-check that a chosen augmentation is genuinely appropriate for the specific data and task.
• Accidentally applying random augmentations to the validation or test set — this introduces unwanted noise into the very evaluation numbers meant to be a stable, reliable measure of generalization.
• Over-augmenting to the point that examples become unrecognizable or unrealistic relative to real deployment data — augmentation should expand the range of realistic variation, not create implausible synthetic distortions.

Interview Relevance

Q: "Why is data augmentation only applied to the training set, never validation or test?" Validation and test sets exist to provide a stable, repeatable measure of generalization performance. Random augmentations would make evaluation non-deterministic — the same example could score differently across runs — undermining the reliability of the very metric they're meant to provide. Augmentation's regularizing benefit comes specifically from exposing the training process to more variation, which is unrelated to how evaluation itself should be measured.

Practice Question

For a dataset of images containing readable street signs, would randomly rotating images by up to 180 degrees be an appropriate augmentation? Why or why not?

Related DL Notes

• Dropout
• Overfitting
• Image Classification