Overfitting (fastai)

Gradient Boosting Overfitting

Overfitting occurs when a machine learning model is too complex and has too many parameters relative to the size of the training data. This can lead the model to fit the training data too closely, resulting in poor generalization to new, unseen data.

Test-Daten sollten eigentlich nur ein EINZIGES MAL genutzt werden, ansonsten besteht immer die Gefahr für Overfitting. Eine gute Erklärung gibt es hier: tweet

How can you tell if your model is overfitting?

One way to tell if your model is overfitting is to evaluate its performance on a separate test set. If the model performs well on the training data but poorly on the test data, this is a sign of overfitting.

Another way to detect overfitting is to plot the model’s performance as a function of the number of training examples. If the model performs well on a small number of training examples but begins to degrade as the number of training examples increases, this is another sign of overfitting.

When does Overfitting start?

Learning curves interpretation

• model is to complex for the data
• tune the model too much with respect to specific validation set

What can you do about overfitting?

One of the best techniques that can be used to address the problem of overfitting in boosting algorithms is early stopping. But there are more techniques:

1. Split the data into training, validation, and test sets: Splitting the data into different sets allows you to evaluate the model’s performance on unseen data. The model should be trained on the training set and evaluated on the validation set. If the model performs well on the validation set, it can then be tested on the test set to get an estimate of its generalization performance.
2. Use regularization: Regularization is a technique that adds a penalty to the model’s complexity to prevent overfitting. It is commonly used in linear and logistic regression, as well as neural networks. There are several types of regularization, such as L1 and L2 regularization, which add a penalty based on the absolute or squared values of the model’s parameters, respectively.