Optimization Tricks: momentum, batch-norm, and more

Deep Learning Crash Course playlist: https://www.youtube.com/playlist?list=PLWKotBjTDoLj3rXBL-nEIPRN9V3a9Cx07

How to Design a Convolutional Neural Network
https://www.youtube.com/watch?v=fTw3K8D5xDs&t=596s

Highlights:
Stochastic Gradient Descent
Momentum Algorithm
Learning Rate Schedules
Adaptive Methods: AdaGrad, RMSProp, and Adam
Internal Covariate Shift
Batch Normalization
Weight Initialization
Local Minima
Saddle Points

References and further reading:

Deep Learning by Ian Goodfellow:
http://www.deeplearningbook.org/

Stochastic gradient descent
https://en.wikipedia.org/wiki/Stochastic_gradient_descent

Adaptive subgradient methods for online learning and stochastic optimization
http://jmlr.org/papers/volume12/duchi11a/duchi11a.pdf

RMSProp Lecture Notes by Geoffrey Hinton
https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf

Adam: A method for stochastic optimization
https://arxiv.org/pdf/1412.6980

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
https://arxiv.org/pdf/1502.03167.pdf

Saddle point
https://en.wikipedia.org/wiki/Saddle_point

Understanding the difficulty of training deep feedforward neural networks
http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf

#deeplearning #machinelearning

Видео Optimization Tricks: momentum, batch-norm, and more канала Leo Isikdogan