Duplicate rows silently give some data points extra "votes" during training — a model trained on a dataset with repeated rows effectively over-weights whatever those rows represent, without you asking it to.
Detecting and Removing Duplicates
import pandas as pd
df = pd.read_csv("transactions.csv")
print(df.duplicated().sum()) # count of exact duplicate rows
df_clean = df.drop_duplicates() # remove exact duplicates, keep first occurrence
# Duplicates based on a subset of columns (e.g. same customer, same day, same amount)
print(df.duplicated(subset=["customer_id", "date", "amount"]).sum())
df_clean = df.drop_duplicates(subset=["customer_id", "date", "amount"], keep="first")Exact Duplicates vs Near-Duplicates
| Type | Example | Detection Approach |
|---|---|---|
| Exact duplicate | Identical row appears twice — often from a data pipeline bug (double-submitted form, re-import) | df.duplicated() |
| Near-duplicate | Same customer logged with slightly different spelling/formatting | Fuzzy matching, or normalize text first (see Data Cleaning) then re-check exact duplicates |
Why This Isn't Always "Just Remove Them"
Not every repeated row is a bug — a customer genuinely making the same $10 purchase twice in one day is two legitimate transactions, not a duplicate. Always check why rows match before deleting: exact duplicates across every column are usually safe to drop; duplicates only on a business key (like transaction_id) are almost always a pipeline error.
Common Mistakes
- Dropping duplicates before splitting into train/test — if a duplicate row lands in both sets, the model effectively "sees" a test example during training, inflating evaluation metrics.
- Using
drop_duplicates()with nosubsetwhen the real business duplicate key is only a few columns (like an ID), missing near-duplicates that differ only in an irrelevant timestamp column.
Interview Relevance
Q: "Why can duplicate rows be a problem even for otherwise clean data?" They silently bias the model toward whatever pattern the duplicated rows represent, and if duplicates span the train/test split, they inflate test performance in a way that won't hold on genuinely new data.
Practice Question
A transactions dataset has duplicate rows on (customer_id, date, amount) but different transaction_id values. Are these safe to drop as duplicates? What would you check first?
