XGBoost

CatBoost

AdaBoost

LightGBM

Gradient Boosting vs Random Forests

<aside> 💡 Algorithm can handle missing values. This is true for all Gradient Boost based models. See also this post.

</aside>

State of Competitive Machine Learning 2022 report: A big surprise for tabular competitions: the reign of XGBoost seems over. While gradient boosting still wins most tabular competitions, LightGBM is now the preferred approach, with CatBoost coming in second. XGBoost is third. (link)

When we consider execution time, LightGBM wins hands down! It is 7 times faster than XGBoost and 2 times faster than CatBoost!

When we consider performance, XGBoost is slightly better than the other two. However, selecting the right boosting technique depends on many factors. Here are some guidelines that help you to choose the right boosting algorithm for your task.

XGBoost vs. LightGBM

XGBoost vs. LightGBM

Guidelines to select the right boosting technique

1. Any boosting technique is much better than decision trees and random forests except for parallelizing the training process.
2. You can start with a basic boosting technique such as AdaBoost, or Gradient Boosting and then you can move to an enhanced technique such as XGBoost.
3. LightGBM and CatBoost are great alternatives to XGBoost.