Pandas for Machine Learning

Pandas is the library you'll spend the most time in before you ever train a model — loading datasets, inspecting them, filtering rows, handling missing values, and shaping data into the format scikit-learn expects.

The First Five Minutes With Any Dataset

import pandas as pd

df = pd.read_csv("loans.csv")

df.head()          # first 5 rows — sanity check the data loaded correctly
df.shape            # (rows, columns)
df.info()            # column names, dtypes, non-null counts — spot missing values fast
df.describe()        # count, mean, std, min, max, quartiles for numeric columns
df.isnull().sum()    # missing values per column

Run these five commands on every new dataset before writing a single line of modeling code — most "surprising" model behavior traces back to something these commands would have shown you (unexpected missing values, wrong dtypes, outlier ranges).

Selecting and Filtering

df["income"]                          # single column -> Series
df[["income", "age"]]                 # multiple columns -> DataFrame

df[df["income"] > 50000]              # filter rows: boolean mask
df[(df["age"] > 30) & (df["income"] > 50000)]   # multiple conditions — use &, not "and"

df.loc[0:5, "income"]                 # label-based selection
df.iloc[0:5, 2]                       # position-based selection

Handling Missing Values and Duplicates

df["income"] = df["income"].fillna(df["income"].median())   # simple imputation
df = df.drop_duplicates()
df = df.dropna(subset=["target_column"])   # drop rows with a missing target — never impute the target

See Missing Values and Missing Value Imputation for when each strategy is appropriate.

Grouping and Aggregating

df.groupby("region")["sales"].mean()          # average sales per region
df.groupby("region").agg({"sales": "sum", "customer_id": "count"})

Preparing Data for scikit-learn

X = df.drop(columns=["churned"])   # everything except the target = features
y = df["churned"]                   # the target column

# scikit-learn accepts DataFrames directly for X, and a Series for y
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Common Mistakes

• Using Python's and/or instead of &/| when combining boolean filters — Pandas requires the bitwise operators, and each condition needs parentheses.
• Modifying a DataFrame slice and getting a SettingWithCopyWarning — use .loc[] for assignment, or explicitly call .copy() when you intend to work on an independent copy.
• Imputing missing values in the target column instead of dropping those rows — you should never fabricate the answer you're trying to predict.

Interview Relevance

Q: "How would you quickly understand a new dataset before modeling it?" df.head(), df.info(), df.describe() and df.isnull().sum() — this combination reveals shape, dtypes, missing values and the general distribution of every numeric column in seconds.

Practice Question

Given a DataFrame df with columns age, salary and department, write Pandas code to find the average salary per department, sorted from highest to lowest.

Related ML Notes

• Pandas DataFrame
• Data Loading in Python
• Data Preprocessing