Feature engineering, data quality and data-centric ML

Garbage in, garbage out — operationalized

A model learns whatever regularities exist in the matrix you hand it. Feature engineering makes the useful regularities easy to find (scale, encode, combine); data quality work removes the fake ones (leakage, duplicates, label noise). Most of the gains in a real project come from this layer, not from swapping algorithms.

Scaling and transforms

Standardization $z=(x-\mu)/\sigma$ and min-max scaling $(x-\min)/(\max-\min)$ put features on comparable scales; the robust scaler uses median and IQR to resist outliers. Heavy right-skewed features (income, counts) benefit from $\log(1+x)$ or Box–Cox/Yeo–Johnson power transforms, which make them more Gaussian and tame outliers. Which models care? Anything built on distances or gradients: kNN (3.4 k-Nearest Neighbors), SVMs (3.10 Support vector machines and kernels), regularized linear models (3.6 Regularization and generalization) and neural networks (4.6 Initialization and normalization). Tree ensembles (3.9 Ensembles: bagging, random forests and gradient boosting) do not. Fit scalers on the training fold only and apply to validation/test.

Encoding categorical features

One-hot encoding creates one binary column per category — fine for low cardinality, wasteful for thousands of categories (sparse, and trees must make many splits). Ordinal codes are only meaningful when the categories are ordered. Target encoding replaces each category with the mean target of that category — powerful but a leakage trap: computed on the full training set it lets each row see its own label. Use out-of-fold estimates with smoothing toward the global mean, $\tilde y_c = \frac{n_c\bar y_c + m\bar y}{n_c+m}$, or CatBoost's ordered statistics (3.9 Ensembles: bagging, random forests and gradient boosting). Hashing handles unbounded vocabularies; learned entity embeddings (a lookup table trained with the model) are the deep-learning answer and the same object as word embeddings (6.3 Word embeddings, 4.10 Autoencoders, embeddings and representation learning).

Dates, text, interactions

Encode cyclical features with $\sin(2\pi t/P)$ and $\cos(2\pi t/P)$ so 23:00 is near 00:00. For time series create lag features and rolling statistics using only past values. Text becomes TF-IDF vectors (6.2 Classical NLP and n-gram language models) or embeddings (9.2 Embeddings deep dive). Products and ratios of features (interactions) let linear models capture what trees find automatically; binning turns a continuous feature into categories when the relationship is step-like. Domain knowledge is the best feature generator: a loan's debt-to-income ratio beats debt and income separately.

Missing values and outliers

Ask why values are missing: completely at random (MCAR), at random given other features (MAR), or because of the value itself (MNAR, e.g. high earners skipping the income field). Impute with mean/median (simple), kNN or iterative model-based imputation (better), and add a missing-indicator column, since missingness is often informative. Tree libraries handle missing values natively (3.8 Decision trees). For outliers, detect with the IQR rule or z-scores for single features and Isolation Forest for multivariate cases; then decide — fix, winsorize (clip), model robustly, or keep — rather than deleting on sight.

Leakage: the silent killer

Leakage is information in training features that will not be available at prediction time, or information from the test set entering training. Three flavours:

1. Target leakage — a feature is a consequence of the label ("received antibiotics" when predicting infection; "account closed" when predicting churn).
2. Train–test contamination — fitting scalers, encoders, imputers, PCA or feature selection on all data before splitting; duplicates or near-duplicates straddling the split.
3. Temporal leakage — random splits on time-ordered data let the model train on the future. Use time-based splits (train on the past, validate on the next window).

The one question that catches most leaks: would this exact value exist, with this exact timestamp, at the moment the prediction is made? The one tool that prevents the rest: put every preprocessing step inside a scikit-learn Pipeline so it is fit on training folds only.

Imbalanced data

With 1% positives, "always negative" scores 99% accuracy. Use metrics that respect the minority class — precision/recall, PR-AUC, F1 at a tuned threshold (3.5 Evaluating models: metrics, validation and the bias–variance trade-off). Then choose a remedy: class weights (cost-sensitive loss), threshold tuning on the validation set, or resampling (undersample the majority, oversample or synthesize minority points with SMOTE — but never resample the validation/test data, and beware that synthetic points can leak across folds). In deep learning, focal loss down-weights easy negatives (4.3 Loss functions catalog).

Feature selection and importance

Selection comes in three flavours: filter (correlation, mutual information with the target — 1.9 Information theory: entropy, cross-entropy and KL divergence), wrapper (recursive feature elimination with a model) and embedded ($\ell_1$ regularization 3.6 Regularization and generalization, tree importances). For importance, prefer permutation importance — shuffle one feature in validation data and measure the metric drop — over impurity-based scores. SHAP values give a principled per-prediction attribution: the Shapley value of feature $i$ is its average marginal contribution over all orderings of features,

and the attributions sum exactly to the prediction minus the baseline. TreeSHAP computes this exactly and fast for tree ensembles; summary and dependence plots show global patterns. Caveat for every importance method: correlated features share or steal credit, so importance is about the model, not about causality.

Data-centric ML and documentation

Andrew Ng's data-centric movement reframes the loop: hold the model fixed and improve the data — fix inconsistent labels (measure inter-annotator agreement), add examples for the slices where error analysis shows failures, remove duplicates and mislabeled points, and write down what the data is (datasheets for datasets, model cards). Version your data (DVC), validate it on ingestion (schema and distribution checks), and keep a curated evaluation set that never leaks into training. In the LLM era this is the mainstream: pretraining corpora are won by filtering and deduplication (8.1 The LLM lifecycle and model landscape), a thousand excellent instruction examples beat a million mediocre ones (8.2 Fine-tuning and PEFT), and your own eval set is the most valuable artifact you build (8.7 Evaluating LLMs).

Where this shows up later

• The end-to-end project applies all of this inside pipelines — 3.14 Project: end-to-end tabular ML with scikit-learn; metrics for imbalance — 3.5 Evaluating models: metrics, validation and the bias–variance trade-off.
• Embeddings as learned encodings — 4.10 Autoencoders, embeddings and representation learning, 6.3 Word embeddings; normalization layers as "scaling inside the network" — 4.6 Initialization and normalization; focal loss — 4.3 Loss functions catalog.
• Data quality for LLM training and evaluation — 8.1 The LLM lifecycle and model landscape, 8.2 Fine-tuning and PEFT, 8.7 Evaluating LLMs; chunking and ingestion as feature engineering for RAG — 9.4 Chunking and ingestion.