In machine learning and statistics, the learning rate is a tuning parameter in an optimization algorithm that determines the step size at each iteration while moving toward a minimum of a loss function.
Since it influences to what extent newly acquired information overrides old information, it metaphorically represents the speed at which a machine learning model “learns”.
In the adaptive control literature, the learning rate is commonly referred to as gain.
In setting a learning rate, there is a trade-off between the rate of convergence and overshooting.
While the descent direction is usually determined from the gradient of the loss function, the learning rate determines how big a step is taken in that direction.
A too high learning rate will make the learning jump over minima but a too low learning rate will either take too long to converge or get stuck in an undesirable local minimum.
