'How neural networks learn' - Part III: The learning dynamics behind generalization and overfitting
In this third episode on "How neural nets learn" I dive into a bunch of academical research that tries to explain why neural networks generalize as wel as they do. We first look at the remarkable capability of DNNs to simply memorize huge amounts of (random) data. We then see how this picture is more subtle when training on real data and finally dive into some beautiful analysis from the viewpoint on information theory.
Main papers discussed in this video:
First paper on Memorization in DNNs: https://arxiv.org/abs/1611.03530
A closer look at memorization in Deep Networks: https://arxiv.org/abs/1706.05394
Opening the Black Box of Deep Neural Networks via Information: https://arxiv.org/abs/1703.00810
Other links:
Quanta Magazine blogpost on Tishby's work: https://www.quantamagazine.org/new-theory-cracks-open-the-black-box-of-deep-learning-20170921/
Tishby's lecture at Stanford: https://youtu.be/XL07WEc2TRI
Amazing lecture by Ilya Sutkever at MIT: https://youtu.be/9EN_HoEk3KY
If you want to support this channel, here is my patreon link:
https://patreon.com/ArxivInsights --- You are amazing!! ;)
If you have questions you would like to discuss with me personally, you can book a 1-on-1 video call through Pensight: https://pensight.com/x/xander-steenbrugge
Видео 'How neural networks learn' - Part III: The learning dynamics behind generalization and overfitting канала Arxiv Insights
Main papers discussed in this video:
First paper on Memorization in DNNs: https://arxiv.org/abs/1611.03530
A closer look at memorization in Deep Networks: https://arxiv.org/abs/1706.05394
Opening the Black Box of Deep Neural Networks via Information: https://arxiv.org/abs/1703.00810
Other links:
Quanta Magazine blogpost on Tishby's work: https://www.quantamagazine.org/new-theory-cracks-open-the-black-box-of-deep-learning-20170921/
Tishby's lecture at Stanford: https://youtu.be/XL07WEc2TRI
Amazing lecture by Ilya Sutkever at MIT: https://youtu.be/9EN_HoEk3KY
If you want to support this channel, here is my patreon link:
https://patreon.com/ArxivInsights --- You are amazing!! ;)
If you have questions you would like to discuss with me personally, you can book a 1-on-1 video call through Pensight: https://pensight.com/x/xander-steenbrugge
Видео 'How neural networks learn' - Part III: The learning dynamics behind generalization and overfitting канала Arxiv Insights
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
!['How neural networks learn' - Part I: Feature Visualization](https://i.ytimg.com/vi/McgxRxi2Jqo/default.jpg)
![Stanford Seminar - Information Theory of Deep Learning](https://i.ytimg.com/vi/XL07WEc2TRI/default.jpg)
![Overcoming sparse rewards in Deep RL: Curiosity, hindsight & auxiliary tasks.](https://i.ytimg.com/vi/0Ey02HT_1Ho/default.jpg)
![Face editing with Generative Adversarial Networks](https://i.ytimg.com/vi/dCKbRCUyop8/default.jpg)
![Arduino Neural Network Robot - Complete How-To!](https://i.ytimg.com/vi/wtNaPLmpy1I/default.jpg)
![Policy Gradient methods and Proximal Policy Optimization (PPO): diving into Deep RL!](https://i.ytimg.com/vi/5P7I-xPq8u8/default.jpg)
![Variational Autoencoders](https://i.ytimg.com/vi/9zKuYvjFFS8/default.jpg)
![The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://i.ytimg.com/vi/ZVVnvZdUMUk/default.jpg)
![Comparing humans with the best Reinforcement Learning algorithms](https://i.ytimg.com/vi/Ol0-c9OE3VQ/default.jpg)
![OpenAI Five: Facing Human Pro's in Dota II](https://i.ytimg.com/vi/0eO2TSVVP1Y/default.jpg)
![On the Foundations of Deep Learning: SGD, Overparametrization, and Generalization](https://i.ytimg.com/vi/l0im8AJAMco/default.jpg)
![Direct and dual Information Bottleneck frameworks for Deep Learning - Tali Tishby](https://i.ytimg.com/vi/TisObdHW8Wo/default.jpg)
![Neural Network that Changes Everything - Computerphile](https://i.ytimg.com/vi/py5byOOHZM8/default.jpg)
![How AlphaGo Zero works - Google DeepMind](https://i.ytimg.com/vi/MgowR4pq3e8/default.jpg)
![An introduction to Reinforcement Learning](https://i.ytimg.com/vi/JgvyzIkgxF0/default.jpg)
![The Rise of Artificial Intelligence through Deep Learning | Yoshua Bengio | TEDxMontreal](https://i.ytimg.com/vi/uawLjkSI7Mo/default.jpg)
![Better than Deep Learning: Gradient Boosting Machines (GBM)](https://i.ytimg.com/vi/9GCEVv94udY/default.jpg)
![How to Detect & Prevent Machine Learning Overfitting](https://i.ytimg.com/vi/3NeRWVul5E0/default.jpg)
!['How neural networks learn' - Part II: Adversarial Examples](https://i.ytimg.com/vi/4rFOkpI0Lcg/default.jpg)
![Tomaso Poggio - Dynamics and Generalization in Deep Neural Networks](https://i.ytimg.com/vi/RnP5iUS0cg4/default.jpg)