How does Batch Normalization Help Optimization?
Batch normalization (BatchNorm) is a widely adopted technique that enables faster and more stable training of deep neural networks. However, despite its pervasiveness, the exact reasons for BatchNorm's effectiveness are still poorly understood.
In this talk, we take a closer look at the underpinnings of the BatchNorm's success. In particular, we examine the popular belief that the root of BatchNorm's effectiveness is due to reduction of an effect called internal covariate shift (ICS). We then explore the connection between BatchNorm, ICS, and the optimization landscape of deep neural networks.
See more at https://www.microsoft.com/en-us/research/video/how-does-batch-normalization-help-optimization/
Видео How does Batch Normalization Help Optimization? канала Microsoft Research
In this talk, we take a closer look at the underpinnings of the BatchNorm's success. In particular, we examine the popular belief that the root of BatchNorm's effectiveness is due to reduction of an effect called internal covariate shift (ICS). We then explore the connection between BatchNorm, ICS, and the optimization landscape of deep neural networks.
See more at https://www.microsoft.com/en-us/research/video/how-does-batch-normalization-help-optimization/
Видео How does Batch Normalization Help Optimization? канала Microsoft Research
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
Why Does Batch Norm Work? (C2W3L06)Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftBatch Normalization (“batch norm”) explainedAuto-Tuning Hyperparameters with Optuna and PyTorch138 - The need for scaling, dropout, and batch normalization in deep learningGroup Normalization12a: Neural NetsResearch in Focus: Deep Learning Research and the Future of AIDeep Dream (Google) - ComputerphileHow optimization for machine learning works, part 1PhD: How to write a great research paperPyTorch at Tesla - Andrej Karpathy, TeslaIllustrated Guide to Transformers Neural Network: A step by step explanationBatch Normalization - EXPLAINED!Deep Learning Basics: Introduction and OverviewAn Introduction to Graph Neural Networks: Models and ApplicationsBatch normalization | What it is and how to implement itBatch Normalization | How does it work, how to implement it (with code)Optimization Tricks: momentum, batch-norm, and more