How to Evaluate a Machine Learning Model: Accuracy, Precision, Recall and F1 Score

Start with the decision, not the highest score

Suppose you are practising with a customer-churn classifier: 1 means the customer left during a defined future period; 0 means they stayed. A positive prediction adds a customer to a review list. A false positive uses review time unnecessarily; a false negative misses someone who actually leaves. Decide which error matters and how many cases the team can review before selecting a metric.

This article focuses on binary classification. These four scores do not evaluate a house-price regression model. For a wider comparison of problem types, use the algorithms guide linked below.

Worked confusion matrix: 100 fictional customers

Teaching example only: the counts below are invented to explain the arithmetic. They are not Softenant student results, client data or a claim about a trained model.

There are 20 actual leavers and 80 stayers. The model flags 20 people, of whom 12 actually leave. Check those totals before calculating percentages: swapping the actual and predicted axes changes the interpretation.

Accuracy, precision, recall and F1: calculations

• Accuracy: (TP + TN) / all cases = (12 + 72) / 100 = 84%. How often was the predicted class correct overall?
• Precision: TP / (TP + FP) = 12 / 20 = 60%. Of the customers flagged, how many actually left?
• Recall: TP / (TP + FN) = 12 / 20 = 60%. Of all actual leavers, how many did the model find?
• F1: 2TP / (2TP + FP + FN) = 24 / 40 = 0.60. This combines precision and recall without using the true-negative count.

The scikit-learn evaluation guide documents classification metrics. Name the positive class and report the underlying counts alongside the scores; a percentage by itself hides the sample size.

Why an 80% baseline still matters

A rule that always predicts “stay” gets 80 of these 100 cases correct. Its accuracy is 80%, but it finds none of the 20 leavers: recall is zero. Precision has a zero denominator because nobody is flagged. A reporting convention may display zero, but the report should say “no positive predictions” rather than imply a measured success rate.

The example classifier improves accuracy by four percentage points over that rule and identifies 12 leavers. That is a more useful explanation than “84% accurate.” It still does not show whether the review programme is worthwhile, whether predictions work next month, or whether the model treats relevant groups adequately.

Reproduce the counts in Python

This self-contained Python example uses only the standard library. It recreates the fictional outcomes; it does not train or benchmark a model.

y_true = [1] * 20 + [0] * 80
y_pred = [1] * 12 + [0] * 8 + [1] * 8 + [0] * 72

tp = sum(t == 1 and p == 1 for t, p in zip(y_true, y_pred))
tn = sum(t == 0 and p == 0 for t, p in zip(y_true, y_pred))
fp = sum(t == 0 and p == 1 for t, p in zip(y_true, y_pred))
fn = sum(t == 1 and p == 0 for t, p in zip(y_true, y_pred))

print([[tn, fp], [fn, tp]])
print((tp + tn) / len(y_true))
print(tp / (tp + fp), tp / (tp + fn))
print(2 * tp / (2 * tp + fp + fn))

Expected output: [[72, 8], [8, 12]], accuracy 0.84, precision and recall 0.6, and F1 0.6. The denominators are non-zero in this example; general-purpose code must explicitly handle empty inputs and undefined metrics. When using scikit-learn, also specify the positive label and averaging method.

A second threshold changes the tradeoff

For the same fictional 100 customers, consider another operating point that flags 35 people: TP = 17, FP = 18, FN = 3 and TN = 62. Its accuracy is 79%, precision approximately 48.57%, recall 85%, and F1 approximately 0.6182.

It finds five more leavers, but sends ten more stayers for review. A slightly higher F1 does not settle whether the larger review list is acceptable. These are illustrative counts, not observed threshold-test results. Use validation data to select a threshold, freeze the choice, then assess it on an untouched test set. The official threshold-tuning guide explains why the model’s score and the action threshold are separate choices.

Checks before trusting an evaluation

1. Prevent leakage. Features must be available at prediction time. Fit imputation, scaling and feature selection on training data, not the test set. Work through the data-cleaning exercise first.
2. Use an appropriate split. Repeated customers may need grouped splits; predicting future periods may require time-based evaluation. A random split is not automatically suitable.
3. Report support. Include the number of positive and negative cases, evaluation dates, label definition and split method. Scores from a few cases are unstable.
4. Inspect the errors. Review permitted examples of false positives and false negatives. Check whether missing fields, different customer segments or recent data cause systematic mistakes.
5. Separate development from assessment. Repeatedly adjusting a model after reading its test results turns that set into development data. Reserve a fresh final assessment when needed.

Questions to practise explaining

Is F1 always better than accuracy? No. It emphasizes positive-class precision and recall but ignores true negatives and does not encode your actual error costs.

What changes with three or more classes? Report per-class results and support. Macro averaging weights classes equally; weighted averaging uses class support. A strong aggregate can still hide a weak class. See the F1 API’s averaging definitions.

What should go into a portfolio? Include the baseline, confusion matrix, chosen threshold, split rationale and limitations. Label simulated numbers as simulated, and publish only results you actually obtained from an authorized dataset.

Next learning steps

Once evaluation is documented, use the deployment release checklist to consider how predictions will be served and monitored. For broader artificial-intelligence course information, see AI Training in Vizag; this guide remains focused on evaluating predictive models.

Build practical job-ready skills

For hands-on learning and guided practice, visit Machine Learning Training in Vizag.