Data Loading in Python

Before any modeling happens, data has to get from wherever it lives — a CSV, a database, an API — into a Pandas DataFrame. Getting this step right (encoding, dtypes, missing markers) saves hours of confusing downstream bugs.

Loading From Common Sources

import pandas as pd

df_csv   = pd.read_csv("data.csv")
df_excel = pd.read_excel("data.xlsx", sheet_name="Sheet1")
df_json  = pd.read_json("data.json")

import sqlite3
conn = sqlite3.connect("database.db")
df_sql = pd.read_sql("SELECT * FROM customers", conn)

Options That Save You From Silent Bugs

df = pd.read_csv(
    "data.csv",
    na_values=["NA", "N/A", "--", "unknown"],   # treat these strings as missing, not text
    dtype={"customer_id": str},                  # force an ID column to stay a string (not lose leading zeros)
    encoding="utf-8",                             # avoid garbled text on non-ASCII data
    low_memory=False                               # avoid dtype-guessing warnings on large mixed-type files
)

Without na_values, a column with the literal text "NA" or "unknown" is read as a normal string category instead of a proper missing value — this silently breaks any downstream missing-value handling.

Loading Large Files

# Read in chunks instead of loading everything into memory at once
chunks = pd.read_csv("huge_file.csv", chunksize=100_000)
total_rows = sum(len(chunk) for chunk in chunks)

# Or load only the columns you actually need
df = pd.read_csv("huge_file.csv", usecols=["age", "income", "churned"])

First Checks After Loading, Every Time

print(df.shape)          # does the row/column count match what you expected?
print(df.dtypes)          # are numeric columns actually numeric?
print(df.isnull().sum())  # where are the missing values, and how many?
print(df.duplicated().sum())  # are there duplicate rows?

Practical Use Cases

• Ingesting raw CSV/Excel exports from business systems for a modeling project
• Pulling training data directly from a SQL database
• Loading only a sample or subset of a very large dataset for fast iteration

Common Mistakes

• Not specifying na_values and later discovering "missing" data was actually being treated as a text category the whole time.
• Loading an entire multi-gigabyte file into memory when only a few columns or a sample were actually needed.
• Not checking df.dtypes immediately — a numeric column read as text will fail (or silently misbehave) in most scikit-learn operations.

Interview Relevance

Q: "How do you handle a dataset too large to fit in memory?" Load selectively — read only needed columns with usecols, process in chunks with chunksize, downcast numeric dtypes, or sample the data for exploratory work before scaling up.

Practice Question

A CSV file has a column where missing values are recorded as the string "?". Write the pd.read_csv call that correctly treats these as missing values (NaN) on load.

Related ML Notes

• Pandas for ML
• Missing Values
• Data Cleaning