Bias and Variance Videos

When training loss is low relative to testing loss we are overfitting to the training data. When training loss is high relative to testing loss we are underfitting. Balancing both cases amounts to finding the best trade-off between bias and variance.

Bias too high

Bias

• Represents how far the model's predictions are from the true values on average
• High bias: The model is too simplistic and underfits the data - it misses important patterns
• Low bias (general goal): The model captures the underlying patterns in the training data well

Variance to high

Variance

• Represents how much the model's predictions change when trained on different datasets
• High variance: The model is too complex and overfits to noise in the training data - it learns patterns that don't generalize
• Low variance (general goal): The model is stable and makes similar predictions across different training sets

Dependencies between bias and variance