Gradient Descent (video from StatQuest) is the workhorse behind most of Machine Learning. When you fit a machine learning method to a training dataset, you're probably using Gradient Descent. It can optimize parameters in a wide variety of settings.

• Gradient Decent in fastai tutorial
• Doing a Gradient Decent step with Pytorch in scratch
• What is the difference between Gradient Descent and Stochastic Gradient Descent?
  • Gradient Descent calculates the gradients on the entire dataset at one go. For Stochastic Gradient Descent (SGD) we calculate gradients on batches. Before moving onto the next batch, we modify our model's parameters based on the gradients. For each iteration through our dataset (which would be called an epoch) Gradient Descent would update the parameters once, SGD would perform as many updates as we have batches.
• Twitter on Gradient Decent, Stochastic Gradient Descent, Batch Gradient Decent,…

Overview

Gradient Decent starts with a guess for the value and then goes into a loop that improves the guess one small step at a time. Although Gradient Decent takes longer than an analytical solution, it is very often used in ML because it can give solutions to situations where there are no analytical solutions, including Deep Neural Networks.

Summarizing Gradient Descent

Very simple example

Find minimum of the loss function

Gradient Decent for one parameter