LLM Pretraining

Pretraining is the massive, foundational training stage where an LLM learns language, facts, and reasoning patterns purely from next-token prediction over enormous amounts of raw, unlabeled text — the single most compute-intensive stage of an LLM's entire lifecycle.

The Setup

Why This Stage Alone Produces a Genuinely Capable Model

To get good at predicting the next word across a truly enormous, diverse corpus of human-written text, a model has to implicitly learn an enormous amount: grammar, facts about the world, reasoning patterns, coding conventions, and more — none of this is explicitly labeled or taught; it all emerges as a side effect of getting good at the single objective of predicting what word comes next, applied at massive scale. This mirrors the "the embedding is a byproduct" insight from Word2Vec, just at vastly greater scale and with much richer emergent behavior.

Scaling Laws — A Brief Note

Research (notably the "Chinchilla" scaling laws) found that model quality depends predictably on the balance between model size (parameters) and training data quantity (tokens) — for a fixed compute budget, there's a specific ratio of parameters to training tokens that tends to produce the best-performing model, and many earlier large models were found to be significantly undertrained relative to their parameter count (echoing the exact same lesson from RoBERTa, just at a much larger scale).

Code — The Core Training Loop, Conceptually

import torch
import torch.nn.functional as F

def pretraining_step(model, token_batch):
    # token_batch: (batch_size, seq_len) -- raw tokenized text, no labels needed
    inputs = token_batch[:, :-1]     # every token except the last
    targets = token_batch[:, 1:]      # every token except the first -- i.e., "the next token"

    logits = model(inputs)              # (batch, seq_len-1, vocab_size)
    loss = F.cross_entropy(logits.reshape(-1, logits.size(-1)), targets.reshape(-1))
    return loss

# The "labels" here are literally just the input sequence, shifted by one position --
# no human annotation required at any point

Common Mistakes

• Assuming pretraining alone produces a helpful, instruction-following assistant — pretraining produces a raw text-continuation model; the additional stages covered in the rest of this category (SFT, alignment) are what shape it into something more directly useful and safe to interact with.
• Underestimating how much data quality (not just quantity) matters — training on low-quality, repetitive, or heavily duplicated web text can meaningfully hurt a model's final quality, which is why substantial effort typically goes into filtering and curating pretraining data.

Interview Relevance

Q: "How can an LLM learn facts, reasoning, and coding ability from a training objective as simple as 'predict the next word'?" To become genuinely good at predicting the next word across an enormous, diverse corpus of human-written text, a model has to implicitly capture a huge amount of underlying structure — grammar, world knowledge, reasoning patterns — since accurately predicting continuations in technical, factual, or logical text requires modeling that structure, even though none of it is explicitly labeled as a separate training signal.

Practice Question

Why is next-token prediction such a convenient pretraining objective specifically for scaling to enormous, diverse, unlabeled datasets?

Related DL Notes

• Next-Token Prediction
• Supervised Fine-Tuning
• What Is an LLM?