• Fastai introduction to Gradient Boosting. Gradient Boosting is generally more accurate then RF but you can absolutly overfit (which you can not with RF as explained here).
  • Boosting vs bagging:
    • Bagging applied to trees → Random Forest
    • Boosting applied to trees → Gradient Boosted Decision Trees

XGBoost is better then Gradient Boost!

Algorithm

Gradient boosting is a boosting algorithm that trains a series of weak learners and combines them to make a strong learner. It works by training a weak learner to make predictions on the training data, and then using the errors of that model as the target for the next model. This process is repeated until a specified number of models has been trained, or until the model performance stops improving.

Understand how Gradient Boost uses the errors of the previous tree to learn → XGBoost book p. 88

Gradient Boost for Regression

Gradient Boost for Classification

Algorithms based on gradient tree boosting:

XGBoost

LightGBM

CatBoost

Application

Boosting (Code)

Gradient Boosting Overfitting

Comparison