ML

How a Neural Network Learns

Backpropagation calculates gradients. Gradient descent uses them to update the parameters.

Gradient descent

The loss surface shows how the loss changes with the parameters. The gradient points toward increasing loss, so gradient descent steps in the opposite direction. A large learning rate can overshoot the minimum.

Original gradient descent diagram comparing learning rates and showing the parameter update
Try the gradient descent demo →

Backpropagation

The forward pass produces a prediction and a loss. Backpropagation applies the chain rule from the output layer back through the hidden layers to find how each weight and bias affects that loss.

Original backpropagation diagram showing forward and backward signals and the loss derivative

Source: Neural Networks Bike Demand, v3. Equation images and diagrams are reproduced from the PowerPoint.