LLM Parameters (and Model Weights)

Parameters (also called weights) are the numeric values inside a neural network that get adjusted during training — collectively, they're what the model "learned." When you see a model described as "7B" or "70B," that's its parameter count, in billions.

What a Parameter Actually Is

Each transformer block contains large matrices of numbers used to transform token representations (in attention and feed-forward layers). Every individual number in every one of these matrices is one parameter. A model with 7 billion parameters has 7 billion individually-learned numeric values, all adjusted during training to collectively minimize next-token prediction error.

Simplified illustration of one tiny piece:
  weight_matrix = [[0.023, -0.451, 0.109, ...],
                    [-0.302, 0.087, 0.664, ...],
                    ...]
  # A real model has billions of these values across many such matrices

Parameter Count and Model Naming

Exact parameter counts for many commercial hosted models (like the largest closed models from major providers) are often not publicly disclosed — treat any specific number you see for those as an estimate or rumor unless the provider has officially confirmed it.

Weight Precision and File Size

Each parameter is stored as a number at some numeric precision — commonly 16-bit floating point for a full-precision model, or lower (8-bit, 4-bit) for a quantized version. Lower precision means a smaller file size and less memory/compute needed to run the model, usually at some small cost to output quality. This is why you'll see the same open-weight model offered in multiple sizes on disk despite having the same parameter count.

Practical Use Case

Parameter count is a rough, imperfect proxy for capability and cost — more parameters generally means better performance on complex tasks, but also higher inference cost and latency. See Model Size for the practical tradeoffs this creates when choosing a model.

Common Mistakes

• Assuming parameter count alone determines quality — training data quality, training technique, and architecture choices all matter significantly too; a well-trained smaller model can outperform a poorly-trained larger one on a given task
• Confusing "parameters" with "training data size" — these are different things: parameters are the model's learned internal values; training data is what it learned from

Interview Relevance

Q: "What does it mean when a model is described as having 7 billion parameters?" — the expected answer: 7 billion individual learned numeric values (weights) inside the model's neural network layers, adjusted during training.

Practice Question

Explain why a 4-bit quantized version of a model uses roughly a quarter of the memory of its 16-bit counterpart, in terms of what a "parameter" actually is.

Related Notes

• Model Size
• LLM Architecture
• LoRA