Data: Collection, Labeling, Splits

Overview

Good data beats clever models. Define what you need, collect representative samples, label consistently, and create honest splits so your evaluation reflects real-world performance.

Data Collection

Start from the problem and metric: what inputs are available at prediction time? what output do you need?
Sources: existing databases/logs, public datasets, web scraping (respect terms), user studies, synthetic data.
Representativeness: capture the diversity you expect in production (devices, lighting, dialects, time of day, edge cases).
Privacy and consent: avoid collecting personally identifiable information (PII) unless necessary; follow policies.

Labeling

Define a labeling guide with examples and non-examples; resolve ambiguous cases.
Use multiple annotators for tricky tasks; measure agreement (Cohen’s kappa/percent agreement).
Spot-check labels and run small pilot rounds before scaling up.
Tools: Label Studio, doccano, CVAT, custom spreadsheets with validation.

Train/Validation/Test Splits

Standard practice: split your dataset into
Train: used to fit the model
Validation (dev): used to tune hyperparameters and compare models
Test: used once at the end to report final metrics
Typical ratios: 70/15/15 or 80/10/10 (depends on data size).
Keep the test set untouched until the end to avoid overfitting to it.

Split Strategies (avoid leakage)

Random split: fine for IID tabular data when records are independent.
Grouped split: keep related items together (e.g., all samples from a user in the same fold).
Time-based split: train on past, validate/test on future (simulates deployment).
Stratified split: preserve class ratios in each split for classification.

Data Leakage (what to avoid)

Leakage happens when information from the future or target leaks into training.
Common leaks:
Splitting after preprocessing (fit scalers/encoders on full data then split) → always split first, then fit on train only.
Deduplicated/near-duplicate items split across train/test.
Using target-derived features (e.g., target mean encoded with global stats without proper technique).

Class Imbalance

When one class is rare, accuracy can be misleading.
Approaches:
Use appropriate metrics (F1, ROC-AUC, PR-AUC).
Resampling (undersample majority, oversample minority like SMOTE).
Class weights in the loss function.

Data Quality and Cleaning

Handle missing values (drop, impute with mean/median/model-based).
Detect outliers; decide whether to cap, transform, or keep.
Standardize formats (timestamps, categories, units).

Versioning and Reproducibility

Save a manifest (dataset version, source commits, query, time range).
Store preprocessing steps (e.g., scikit-learn Pipelines) and random seeds.
Tools: DVC, MLflow, Weights & Biases Artifacts.

Example

Spam classifier on emails: - Collect: export emails and labels (spam/ham) from last 12 months. - Labeling: verify a sample by two annotators with a simple guide. - Splits: time-based split (train on months 1–9, val on 10–11, test on month 12). - Preprocessing: tokenize text; fit TF-IDF on train only; apply to val/test. - Evaluation: use ROC-AUC and F1 due to class imbalance.

Checklist

Clearly define inputs, outputs, and evaluation metric
Collect representative, consented data; document sources
Write a labeling guide; measure annotator agreement
Split first; fit preprocessing only on train
Use stratified/grouped/time-based splits when needed
Watch for leakage; double-check duplicates and time boundaries
Choose metrics appropriate for class balance
Version datasets, labels, and preprocessing artifacts

Resources

scikit-learn: model_selection (train_test_split, StratifiedKFold, GroupKFold)
Label Studio (labelstud.io), Doccano (doccano.herokuapp.com), CVAT (cvat.org)
DVC (dvc.org), MLflow (mlflow.org)