Exploratory Data Analysis (EDA)

Purpose

Before modeling, you need to understand the shape, quality, and signal in the data. EDA prevents wasted time training models on broken, leaky, or biased datasets.

Core Goals

Peek at structure (rows, columns, types)
Inspect distributions & ranges
Identify missingness patterns
Detect class imbalance or skew
Spot obvious leakage or impossible values
Generate initial feature ideas
Decide if you have enough signal to proceed

Quick Workflow (Iterative)

Load -> Profile -> Question -> Visualize -> Note Issues -> Clean/Fix -> Repeat

You rarely do EDA once; you loop as new anomalies appear.

Essential Commands (pandas)

import pandas as pd

df = pd.read_csv('data.csv')
print(df.shape)          # (rows, cols)
print(df.dtypes)         # data types
print(df.head())         # first rows
print(df.isna().mean())  # missing ratio per column
print(df.describe())     # numeric summary
print(df['target'].value_counts(normalize=True))  # class balance

Visual Checks (Pick a Few First)

Question	Tool
Distribution shape?	Histogram / KDE
Outliers or extreme?	Boxplot / IQR
Correlation hints?	Correlation matrix heatmap
Target vs feature?	Scatter / violin / bar
Time drift?	Line plot over date

Interpreting What You See

Highly skewed numeric: consider log transform or robust scaling.
Many missing values: decide drop vs impute (justify why).
Single category dominating: maybe collapse rare categories into "Other".
Impossible timestamps or future data: potential leakage.
Target present inside a feature (encoded): leakage; remove or recompute.

Common Pitfalls

Pitfall	Fix
Doing EDA after modeling	Always profile before feature engineering
Treating all outliers as errors	Confirm if they are legitimate domain extremes
Using test set during EDA	Restrict deep peeking to train/validation only
Over-tuning thinking from validation set	Minimize repeated manual adjustments using val feedback

Minimal Visualization Snippet

import seaborn as sns, matplotlib.pyplot as plt
sns.histplot(df['age'], kde=True)
plt.title('Age distribution')
plt.show()

From EDA to Features

Write down each potential feature idea with a short rationale ("hour_of_day may correlate with demand peaks"). Only implement after you list several so you avoid tunnel vision.

Mini Exercise

Load a public dataset (e.g., Titanic).
List 5 data quality issues.
Produce 2 plots exploring target relationships.
Propose 3 feature ideas.

Checklist

Shape & types inspected
Missingness summarized
Target distribution understood
Outliers reviewed (not blindly removed)
Leakage indicators checked
Initial feature list drafted

Resources

pandas profiling (ydata-profiling) – automated overview (use judiciously)
Seaborn documentation
"Rules of Machine Learning" (Google) – early data focus