Practical guidance on choosing how many layers a network should have — building on the depth/width tradeoff from Neural Network Architecture.
A Practical Starting Strategy
Rather than guessing an ideal depth from scratch, a common practical approach: start with a relatively shallow network (fewer layers), establish a working baseline, then incrementally add depth while monitoring validation performance — stopping once additional depth stops producing meaningful improvement, or starts hurting (a sign of diminishing returns, training difficulty, or overfitting relative to the available data).
Diagnosing From Symptoms
| Symptom | Likely Interpretation |
|---|---|
| Both training and validation performance are poor (underfitting, see Underfitting) | The network may be too shallow to capture the task's complexity — try adding depth |
| Training performance is excellent, validation performance is much worse (overfitting) | Adding more depth is unlikely to help, and may make overfitting worse — consider regularization or more data instead |
| Training becomes unstable or loss doesn't decrease as depth increases | Very deep networks without adequate residual connections/normalization can suffer vanishing gradients (see Vanishing Gradient Problem) — consider architectural fixes before assuming depth itself is wrong |
Leaning on Established Architectures
For well-studied task types (image classification, standard NLP tasks), starting from a proven, established architecture's depth (a specific ResNet variant, a specific Transformer configuration) — rather than searching depth from scratch — is often far more efficient than independently rediscovering a good depth, since substantial community research has already explored this tradeoff extensively for these common cases.
Code — A Simple Depth Sweep
results = {}
for num_layers in [2, 4, 6, 8]:
model = build_model(num_layers=num_layers)
train(model, train_loader, epochs=20)
val_acc = evaluate(model, val_loader)
results[num_layers] = val_acc
print(f"depth={num_layers}: val_accuracy={val_acc:.4f}")
best_depth = max(results, key=results.get)Common Mistakes
- Adding depth as a default response to poor performance, without first checking whether the actual problem is underfitting versus overfitting — more depth only helps with the former; it can make the latter worse.
- Increasing depth without adding the supporting techniques (residual connections, normalization) that make very deep networks actually trainable — simply stacking more layers without these can make training harder, not better, regardless of how much capacity the extra depth theoretically adds.
Interview Relevance
Q: "A model is underfitting — both training and validation accuracy are low. Would adding more layers be a reasonable first thing to try?" Yes, potentially — underfitting suggests the model may lack sufficient capacity to capture the task's true complexity, and increasing depth (or width) is a reasonable lever to try, alongside checking for other underfitting causes (excessive regularization, insufficient training, poor features) as covered in Underfitting.
Practice Question
Why is leaning on an established, well-studied architecture's depth often a more efficient starting point than independently searching depth from scratch for a common task type like image classification?
