Data Cleaning for Machine Learning: Preparing Better Datasets Before Model Training

What a useful cleaning workflow should produce

Data cleaning for machine learning is not a command that makes every empty cell disappear. Its output should be a traceable dataset: you can explain what each row represents, why a value changed, which rows were excluded, and whether the remaining data still represents the problem.

This guide is a data-quality worksheet, not a complete feature-engineering or model-selection tutorial. Keep an unchanged source copy and create a working version. Use only data you are permitted to process; the example below contains fictional records and no personal information.

A five-row example with four different issues

Assume each row should describe one monthly customer snapshot. The contract says monthly_spend is a non-negative amount in one stated currency. Here, e01 appears twice because the same source event was copied twice. A real repeated purchase would not automatically be a duplicate.

This tiny fixture is suitable for testing cleaning rules, not for training or estimating model accuracy. There is no target label and too little data for a credible performance claim.

Make one explicit decision per issue

• Copied event: verify that the two e01 rows are the same observation, then retain one copy in the working dataset. Keep the original file and log one removed copy. Do not deduplicate by customer ID alone when multiple time periods are legitimate.
• Missing amount: map the agreed token “unknown” to a missing numeric value. Do not convert it to zero: zero spending and unknown spending mean different things.
• Invalid negative amount: flag -50 for source review because it violates this fixture’s contract. If the real field includes refunds, that contract may be wrong. Do not silently take an absolute value or overwrite it with an average.
• Inconsistent text: trim surrounding whitespace and apply an approved city-label mapping. The three Vizag spellings can share a normalized category while the original text remains available for audit.
• Large amount: flag 75000 for review if appropriate, but do not remove it just because it is much larger than the other values. It may be a valid high-spend customer.

After removing the one confirmed duplicate copy, there are four rows. With “unknown” mapped to missing, there is one missing numeric amount and one negative amount awaiting review. Those are separate conditions. This is a quality report, not a declaration that the dataset is ready for modelling.

Profile the data without losing the evidence

For a pandas workflow, inspect column types, distinct labels, missing counts and duplicate keys before deciding how to transform them. Use isna() to detect missing values rather than equality checks against NaN; the pandas missing-data guide describes the differences between missing-value representations.

If numeric parsing fails, retain the source text and record the failure. The string “1,200” may represent a thousands separator, but other exports use commas for decimal values. Confirm the locale and currency rather than stripping punctuation indiscriminately. Likewise, an empty string or a special token such as “unknown” needs an explicit normalization rule before counting missing values.

Keep four fields in the change log: record key, original value, applied rule and reason. For this exercise, an entry could read “e01 duplicate copy removed after confirming identical source event.” In production, protect that log with the same care as the underlying data and retain only what the audit requires.

Separate fixed checks from learned preprocessing

A predefined validity rule, such as checking a required identifier or an agreed non-negative field, is different from learning a median or a scaling parameter from data. Establish the prediction time and split strategy before making data-driven modelling choices. If the same entity appears repeatedly, preserve the group or time boundary needed for a realistic evaluation.

Learn imputation values, scaling parameters, category encodings and feature-selection choices from the training partition. Apply the fitted transformation to validation or test data without refitting it there. During cross-validation, learned preprocessing belongs inside the training portion of each fold. A Pipeline can help enforce this; the scikit-learn leakage guidance shows the underlying risk.

A pipeline does not make an invalid feature safe. A cancellation reason recorded after a customer has left still leaks the answer into a churn model, even if it passes through a pipeline. For each candidate feature, ask: “Would this value exist at the instant I need to predict?” Exclude unavailable future information from the model inputs.

Choose missing-value and outlier treatment deliberately

Investigate why a value is absent: a sensor failure, an optional form field and “not applicable” are different mechanisms. A training-set median can be a starting option for some numeric features, but it is not a universal fix. Compare candidate treatments using development data and document the assumptions.

Removing every row with any missing field can shrink the dataset and disproportionately remove particular groups. Report retained and excluded counts and inspect meaningful segments. For genuine extreme values, consider whether the model or transformation can handle them before deciding on exclusion. Any data-derived clipping threshold must also be learned without using the held-out test distribution.

Acceptance checks before model training

1. Confirm the row grain, required keys, units, date parsing and allowed categories.
2. Reconcile starting rows, confirmed duplicate copies, exclusions and remaining rows; keep unresolved records visible in the report.
3. Check missing values and invalid ranges again after transformations. A successful script run is not evidence that every field is valid.
4. Ensure the target aligns with the correct row and prediction horizon. Drop identifier columns from model inputs when they have no justified predictive role, while retaining keys for controlled joins and audits.
5. Check for entity, duplicate-event and time overlap across evaluation boundaries.
6. Save the rule version, source version, split definition and fitted preprocessing with the experiment.

Turn the worksheet into a small portfolio exercise

Create the five fictional rows, write the rules, and show the expected four-row reconciliation. Add a sixth test row with a new city label: document whether your system accepts, maps or flags it. Add an invalid number token and show that it becomes a visible quality issue rather than disappearing silently.

Next, use a larger authorized, labelled dataset and compare a documented baseline with one revised cleaning choice. Record only measured outcomes and explain any change in sample composition. Use the worked model-evaluation guide to interpret classification results. A cleaner-looking table alone does not prove better predictions.

Related learning path

The feature-engineering guide below covers the next stage: turning suitable data into model inputs. For broader artificial-intelligence course information, visit AI Training in Vizag. The practical focus here remains trustworthy data preparation for predictive models.

Build practical job-ready skills

For hands-on learning and guided practice, visit Machine Learning Training in Vizag.