Data cleaning is the broad pass over raw data to fix structural problems — wrong data types, inconsistent text formatting, invalid values — before any deeper preprocessing like imputation or encoding happens.
The Cleaning Checklist
import pandas as pd
df = pd.read_csv("customers.csv")
# 1. Check dtypes — numbers stored as text are invisible bugs
print(df.dtypes)
df["income"] = pd.to_numeric(df["income"], errors="coerce") # invalid values become NaN
# 2. Standardize inconsistent text categories
df["city"] = df["city"].str.strip().str.lower()
df["city"] = df["city"].replace({"delhi ncr": "delhi", "new delhi": "delhi"})
# 3. Check for invalid/impossible values
print(df[df["age"] < 0]) # negative ages — a data entry error
print(df[df["age"] > 120]) # implausible ages
# 4. Check column names
df.columns = df.columns.str.strip().str.lower().str.replace(" ", "_")Why "lower/upper/delhi ncr" Text Inconsistency Matters
To a human, "Delhi", "delhi ", and "New Delhi" obviously mean the same thing. To a model — especially after one-hot encoding — these become three completely separate, unrelated categories, silently fragmenting your data and weakening any pattern the city feature could have provided.
Practical Use Cases
- Cleaning survey data with inconsistent free-text entries
- Fixing dtype issues after loading a CSV with mixed formatting
- Catching data-entry errors (negative ages, impossible dates) before they corrupt a model silently
Common Mistakes
- Assuming a column labeled numeric is actually numeric — currency symbols, commas, or stray text ("N/A" mixed into a numeric column) silently make a column an
objectdtype. - Cleaning text case-sensitively only halfway (fixing "Delhi"/"delhi" but missing "DELHI") — always normalize case and whitespace together.
Interview Relevance
Q: "How would you catch a data entry error like a negative age in a dataset?" Range/sanity checks after loading — df[df['age'] < 0] or df.describe()'s min/max — combined with domain knowledge about what a plausible value range actually is.
Practice Question
A "country" column contains the values "India", "india", "INDIA " and "Bharat" all meaning the same country. Write the Pandas code to normalize all of these into a single consistent value.
