Weights and bias are the numbers a neural network actually learns during training — everything else about the network (its architecture, its activation functions) is fixed by design before training even starts.
What Weights Represent
Each weight controls how strongly one specific input (or one neuron's output, for weights between hidden layers) influences the next neuron's computation. A large positive weight means that input strongly pushes the neuron toward firing; a large negative weight means it strongly suppresses firing; a weight near zero means that particular connection barely matters for this neuron's decision.
What Bias Represents
The bias \(b\) shifts the neuron's decision threshold independent of the inputs — it lets the neuron fire (or not) even when all inputs are zero. Without a bias term, the decision boundary \(\mathbf{w}^\top\mathbf{x}=0\) is forced through the origin, a real geometric restriction. With a bias, the boundary can sit anywhere in the input space.
Numerical Example — Why Bias Matters Geometrically
Consider a single input, single neuron with step activation: without bias, \(z=wx\), and the decision boundary is always at \(x=0\) regardless of \(w\). With bias, \(z=wx+b\), the boundary sits at \(x=-b/w\) — any point, chosen by learning \(b\). This one extra scalar parameter meaningfully expands what the neuron can represent.
Weight Matrices for a Full Layer
A layer with \(n\) inputs and \(m\) output neurons has a weight matrix \(\mathbf{W}\) of shape \((m,n)\) — one row per output neuron, one column per input — and a bias vector \(\mathbf{b}\) of shape \((m,)\), one bias per output neuron:
Code — Inspecting a Layer's Weights and Bias
import torch.nn as nn
layer = nn.Linear(in_features=4, out_features=3)
print(layer.weight.shape) # torch.Size([3, 4]) -- (out_features, in_features)
print(layer.bias.shape) # torch.Size([3]) -- one bias per output neuron
print(layer.weight) # randomly initialized, will be updated during training
print(layer.bias)Where Initial Values Come From
Weights are never initialized to all zeros in practice (this would make every neuron in a layer compute an identical output and gradient, a failure mode called the "symmetry problem" — every neuron would learn the exact same thing forever). Instead, weights are initialized to small random values, often drawn from a scaled normal or uniform distribution (Xavier/He initialization). Biases are commonly initialized to zero, since the symmetry problem doesn't apply to them the same way.
Common Mistakes
- Initializing all weights to zero (or the same constant) — every neuron in a layer would then receive identical gradients and stay identical forever, effectively wasting the layer's capacity.
- Forgetting that a weight matrix's shape convention (\((\text{out},\text{in})\) vs \((\text{in},\text{out})\)) differs across frameworks — always check
.shapewhen debugging.
Interview Relevance
Q: "Why can't you initialize all of a neural network's weights to zero?" Every neuron in a layer would compute the exact same output (since they'd receive the same inputs and use the same, identical weights), and during backpropagation they'd receive identical gradients too — the "symmetry problem." The layer would never differentiate its neurons from each other, wasting its representational capacity regardless of how long training runs.
Practice Question
A layer has 128 input features and 64 output neurons. What are the shapes of its weight matrix and bias vector, and how many total learnable parameters does this one layer have?
