Evolving Normalization-Activation Layers
Normalization and activation layers have seen a long history of hand-crafted variants with various results. This paper proposes an evolutionary search to determine the ultimate, final and best combined normalization-activation layer... in a very specific setting.
https://arxiv.org/abs/2004.02967
Abstract:
Normalization layers and activation functions are critical components in deep neural networks that frequently co-locate with each other. Instead of designing them separately, we unify them into a single computation graph, and evolve its structure starting from low-level primitives. Our layer search algorithm leads to the discovery of EvoNorms, a set of new normalization-activation layers that go beyond existing design patterns. Several of these layers enjoy the property of being independent from the batch statistics. Our experiments show that EvoNorms not only excel on a variety of image classification models including ResNets, MobileNets and EfficientNets, but also transfer well to Mask R-CNN for instance segmentation and BigGAN for image synthesis, outperforming BatchNorm and GroupNorm based layers by a significant margin in many cases.
Authors: Hanxiao Liu, Andrew Brock, Karen Simonyan, Quoc V. Le
Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Видео Evolving Normalization-Activation Layers канала Yannic Kilcher
https://arxiv.org/abs/2004.02967
Abstract:
Normalization layers and activation functions are critical components in deep neural networks that frequently co-locate with each other. Instead of designing them separately, we unify them into a single computation graph, and evolve its structure starting from low-level primitives. Our layer search algorithm leads to the discovery of EvoNorms, a set of new normalization-activation layers that go beyond existing design patterns. Several of these layers enjoy the property of being independent from the batch statistics. Our experiments show that EvoNorms not only excel on a variety of image classification models including ResNets, MobileNets and EfficientNets, but also transfer well to Mask R-CNN for instance segmentation and BigGAN for image synthesis, outperforming BatchNorm and GroupNorm based layers by a significant margin in many cases.
Authors: Hanxiao Liu, Andrew Brock, Karen Simonyan, Quoc V. Le
Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Видео Evolving Normalization-Activation Layers канала Yannic Kilcher
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftGroup Normalization (Paper Explained)Multimodal Neurons in Artificial Neural Networks (w/ OpenAI Microscope, Research Paper Explained)CARNIVAL SCAM SCIENCE- and how to winMD vs. Machine: Artificial intelligence in health careWhy We Use the Sigmoid Function in Neural Networks for Binary ClassificationAgent57: Outperforming the Atari Human BenchmarkReBeL - Combining Deep Reinforcement Learning and Search for Imperfect-Information Games (Explained)It's Rocket Science! with Professor Chris BishopAn introduction to Reinforcement LearningBatch Normalization (“batch norm”) explainedEvolving Normalization-Activation LayersDeep Networks Are Kernel Machines (Paper Explained)Why Does Batch Norm Work? (C2W3L06)Feedback Transformers: Addressing Some Limitations of Transformers with Feedback Memory (Explained)OpenAI CLIP: ConnectingText and Images (Paper Explained)The Richness of TimeDream to Control: Learning Behaviors by Latent ImaginationBatch NormalizationNFNets: High-Performance Large-Scale Image Recognition Without Normalization (ML Paper Explained)