Probability Distributions

Not each algorithms returns output that can be translated into a probability. In this video we learn about probability calibration and calibrated classifiers in Scikit-Learn and Python.

Tags: CalibratedClassifierCV, calibration

https://www.youtube.com/watch?v=wN4N7IBk16A

Importance of Model Calibration !!!

Model calibration refers to how well a model's predicted probabilities match the actual observed frequencies of events. It can not be seen in the Classification report

Let's say your model makes predictions about whether emails are spam (1) or not spam (0). When your model says:

• "I'm 80% confident this is spam" - it should actually be spam about 80% of the time
• "I'm 90% confident this is spam" - it should actually be spam about 90% of the time

Looking at your image, both models A and B have similar performance metrics (precision, recall, F1-score), with B being slightly better. However, these metrics don't tell us about calibration.

A poorly calibrated model might say:

• "I'm 90% confident this is spam" - but it's actually spam only 60% of the time
• "I'm 60% confident this is spam" - but it's actually spam 90% of the time

This is problematic because:

1. In real applications, we often need to make decisions based on probability thresholds