RMSProp Optimization from Scratch in Python

In this video I will show you how the RMSprop algorithm work for stochastic gradient descent by going through the formula and a Python implementation.

Code: https://github.com/yacineMahdid/artificial-intelligence-and-machine-learning

# Table of Content

- Introduction : 0:00
- Algorithm explanation : 0:45
- Python implementation: 2:31
- Conclusion: 7:30

This algorithm proposed Geoff Hinton in order to palliate to Adagrad shortcoming with vanishing learning rate ([http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf](http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf)). Here is the brief wikipedia explanation on the algorithm : "RMSProp (for Root Mean Square Propagation) is also a method in which the learning rate is adapted for each of the parameters. The idea is to divide the learning rate for a weight by a running average of the magnitudes of recent gradients for that weight."

The algorithm work by having a running average component that is being calculated based on the gradient and the past average. This component is then used during the update of the parameters to weight the update based on the gradient. This way radical shift of gradient are smoothed out because of the running average that is weighting at 90% what happened in the past.

You can also check out this blog post for more information on this algorithm and other that are related: [https://ruder.io/optimizing-gradient-descent/index.html#rmsprop](https://ruder.io/optimizing-gradient-descent/index.html#rmsprop)

# Credit

Music was taken from Youtube music and the content of this video was made possible by a mix of wikipedia, [ruder.io](http://ruder.io) blog post and Geoff Hinto slides.
----
Join the Discord for general discussion: https://discord.gg/QpkxRbQBpf

----
Follow Me Online Here:

Twitter: https://twitter.com/CodeThisCodeTh1
GitHub: https://github.com/yacineMahdid
LinkedIn: https://www.linkedin.com/in/yacine-mahdid-809425163/
Instagram: https://www.instagram.com/yacine_mahdid/
___

Have a great week! 👋

Видео RMSProp Optimization from Scratch in Python канала Deep Learning Explained with Yacine