Semantic segmentation labels every pixel with a class — "road," "sky," "car," "person" — but makes no distinction between separate objects of the same class; two different cars are both simply labeled "car," with no way to tell them apart from the output alone.
Numerical Example
A tiny 4×4 image containing two cars and sky, semantically segmented (using class indices: 0=sky, 1=car):
Both cars share the exact same class label (1) — the output has no notion of "this is car A" versus "this is car B," even though a human looking at the original image could clearly see two separate vehicles.
Where Semantic Segmentation Is Sufficient
Tasks where counting or distinguishing individual objects doesn't matter — e.g. self-driving car "drivable surface" detection (all road pixels matter equally, regardless of how many separate road segments are visible), or medical image segmentation of a single organ type (see U-Net's common application) where instance separation typically isn't needed.
Code
import torch
import torchvision.models.segmentation as seg_models
model = seg_models.fcn_resnet50(weights='DEFAULT')
model.eval()
x = torch.randn(1, 3, 224, 224)
output = model(x)['out']
print(output.shape) # torch.Size([1, 21, 224, 224]) -- 21 classes, one prediction per pixel
predicted_classes = output.argmax(dim=1)
print(predicted_classes.shape) # torch.Size([1, 224, 224]) -- final per-pixel class mapCommon Mistakes
- Using semantic segmentation when a task actually requires counting or distinguishing individual object instances — this calls for instance segmentation (next note) instead.
Interview Relevance
Q: "If an image has three overlapping people, what would a semantic segmentation model's output look like?" Every pixel belonging to any person would be labeled with the same "person" class — there would be no way to distinguish which pixels belong to which specific individual from the segmentation output alone; that distinction requires instance segmentation.
Practice Question
Why might semantic segmentation alone be sufficient for a "is this pixel drivable road or not" self-driving task, but insufficient for "how many pedestrians are in this scene"?
