pandas, data wrangling and plotting

DataFrames and Series

A DataFrame is a table: an ordered collection of columns (Series), each with one dtype, sharing a row index. Think of it as a dictionary of NumPy arrays with labels, plus a large library of table operations. Loading is one line — pd.read_csv("train.csv"), pd.read_parquet("events.parquet") (Parquet is columnar, typed and 5–10× smaller than CSV; prefer it for anything you save yourself) — and the first five minutes with any dataset are always the same:

import pandas as pd
df = pd.read_csv("train.csv")
df.shape                      # (891, 12)
df.head()                     # first rows: do the columns mean what you think?
df.info()                     # dtypes and non-null counts: what is missing, what was parsed wrong
df.describe()                 # count/mean/std/min/quartiles/max for numeric columns
df["Embarked"].value_counts(dropna=False)   # category frequencies, including NaN
df.isna().mean().sort_values(ascending=False)   # fraction missing per column

info() is where you discover that a numeric column was read as object (text) because one row contains "N/A", that a date is a string, or that an id column is numeric and about to be fed to a model as if it were a quantity.

Selecting: loc, iloc and masks

df.loc[rows, cols] selects by label, df.iloc[rows, cols] by position. df.loc[df["Age"] > 30, ["Name", "Age"]] combines a boolean mask with a column list. Combine conditions with &, |, ~ and parentheses — Python's and/or do not work element-wise. To modify a subset, always assign through a single .loc: df.loc[mask, "Age"] = 30. Writing df[mask]["Age"] = 30 (chained indexing) may modify a temporary copy and silently do nothing; recent pandas versions warn about exactly this. df.query("Age > 30 and Sex == 'female'") is a readable alternative for complex filters.

Split–apply–combine: groupby

Most summary questions are "for each group, compute something":

df.groupby("Pclass")["Survived"].mean()                  # survival rate by class
df.groupby(["Pclass", "Sex"]).agg(n=("Survived", "size"),
                                  rate=("Survived", "mean"),
                                  age_med=("Age", "median"))
df.pivot_table(values="Survived", index="Pclass", columns="Sex", aggfunc="mean")
df["fare_rank_in_class"] = df.groupby("Pclass")["Fare"].rank(pct=True)   # transform keeps row count

agg reduces each group to one row; transform returns a result aligned to the original rows (useful for group-wise normalization, a common feature-engineering move in 3.13 Feature engineering, data quality and data-centric ML). A pivot table is a groupby reshaped into a grid. Behind the scenes, pandas is doing the vectorized operations of 2.2 NumPy and vectorized thinking once per group, which is why a groupby over a million rows takes milliseconds.

Joining tables

pd.merge(left, right, on="user_id", how="left") joins on a key. how matters: inner keeps only matches, left keeps every left row (filling NaN where the right side is missing), outer keeps everything. Always check row counts before and after a merge: a count that grows means duplicate keys on the right side created a many-to-many explosion — pass validate="one_to_one" or "many_to_one" to make pandas raise instead. pd.concat([a, b]) stacks tables vertically (more rows) or, with axis=1, horizontally.

Tidy data and reshaping

A tidy table has one variable per column, one observation per row, one value per cell. Models want tidy data; humans often record wide data (one column per month). df.melt(id_vars=["store"], var_name="month", value_name="sales") goes wide → long (tidy); df.pivot(index="store", columns="month", values="sales") goes back. Tidying first makes grouping, plotting and joining uniform.

Missing values, categories and dates

Missingness is information. First ask why values are missing (never recorded? not applicable? a sensor failure correlated with the target?) and record the answer; then choose: drop rows (if few and random), drop the column (if mostly empty), impute (median for numeric, most-frequent or a "missing" category for categorical) and add an indicator column Age_missing so the model can use the fact of missingness. Fit imputation statistics on the training split only, otherwise information leaks from validation data (3.5 Evaluating models: metrics, validation and the bias–variance trade-off, 3.13 Feature engineering, data quality and data-centric ML).

Categorical text columns need encoding before modelling: pd.get_dummies(df["Embarked"], prefix="emb") one-hot encodes; astype("category") stores them compactly and lets tree libraries handle them natively (3.9 Ensembles: bagging, random forests and gradient boosting). Dates parse with pd.to_datetime; then .dt.year, .dt.dayofweek, .dt.hour and differences between dates become numeric features. Timestamps are also how you must split time-series data — train on the past, validate on the future — to avoid the most common leakage of all.

Plotting: the five EDA plots

matplotlib draws anything; seaborn adds statistical defaults on top. Learn the object-oriented interface — fig, ax = plt.subplots(), then ax.hist, ax.scatter, ax.set_xlabel — because it scales to grids of subplots and is what library code returns. Five plots answer most questions:

import matplotlib.pyplot as plt, seaborn as sns
fig, axes = plt.subplots(1, 3, figsize=(14, 4))
axes[0].hist(df["Fare"].dropna(), bins=50); axes[0].set_xlabel("Fare"); axes[0].set_yscale("log")
sns.barplot(data=df, x="Pclass", y="Survived", hue="Sex", ax=axes[1])
sns.heatmap(df.select_dtypes("number").corr(), cmap="coolwarm", center=0, ax=axes[2])
fig.tight_layout(); fig.savefig("eda.png", dpi=150)

A reproducible EDA checklist

1. Shape, dtypes, head; fix parsing (dtypes, dates, sentinels).
2. Missingness per column and its pattern; decide and document a policy.
3. Target distribution: imbalance, range, outliers.
4. Per-feature distributions; candidate transforms.
5. Feature–target relationships; feature–feature redundancy.
6. Leakage audit: anything that would not be known at prediction time (future timestamps, post-outcome fields, ids that encode the answer).
7. Write the findings down in the notebook before modelling — they drive 3.13 Feature engineering, data quality and data-centric ML.

Where this shows up later

• Every project lesson starts with this EDA — 3.14 Project: end-to-end tabular ML with scikit-learn, 5.10 Project: train and deploy an image classifier, 8.13 Project: fine-tune a small LLM with LoRA and ship a chat app.
• Encoding, scaling, imputation and leakage are formalized in 3.13 Feature engineering, data quality and data-centric ML; validation strategy in 3.5 Evaluating models: metrics, validation and the bias–variance trade-off.
• Plots of loss curves and metrics in training use the same matplotlib interface — 4.8 Training dynamics and debugging.
• Hugging Face datasets objects convert to and from pandas — 2.5 Tooling: Git, GPUs, experiments and the Hugging Face Hub.