The very first step is to make an initial prediction. This prediction can be anything, but by default it is 0.5, regardless of whether you are doing Classification or Regression.
Just like the ordinary Gradient Boost, XGBoost fits a Regression Tree to the residuals:
The difference is, that XGBoost uses a special tree and not a regular, off-the-shelf tree. Let’s call it XGBoost Tree.
Each tree start out as a single leaf and all of the residuals go to the leaf.
Now we calculate a Quality Score (Similarity Score) for the residuals. For now, let $\lambda=0$. Here the result is Similarity=4.
