stochastic gradient descent — Jemoka Knowledge Base

gradient descent makes a pass over all points to make one gradient step. We can instead approximate gradients on a minibatch of data. This is the idea behind stochastic-gradient-descent.

\begin{equation} \theta^{t+1} = \theta^{t} - \eta \nabla_{\theta} L(f_{\theta}(x), y) \end{equation}

this terminates when theta differences becomes small, or when progress halts: like when \theta begins going up instead. we update the weights in SGD by taking a single random sample and moving weights to that direction. while not_converged(): for x,y in 1...n: theta = theta - alpha*(loss(x,y).grad()) In theory this is an approximation of gradient descent; however, Neural Networks works actually BETTER when you jiggle a bit. batch gradient descent batch gradient descent does it over the entire dataset by summing loss term over the entire dataset, which is fine but its slow: you are summing over your whole (possibly TBs) dataset, rerunning inference, and then taking a tiny step. This is rather slow. stochastic gradient descent gives choppy movements because it does one sample at once.

\begin{equation} \theta := \theta - \frac{\alpha}{B} \sum_{k=1}^{B} \nabla_{\theta} J^{(j_{k})} \left(\theta\right) \end{equation}

where B is the batch size and \alpha is the learning rate. mini-batch gradient mini-batches helps take advantage of both by training over groups of m samples See