Robert Nowak - What Kinds of Functions Do Neural Networks Learn?
Presentation given by Robert Nowak on 13th October in the one world seminar on the mathematics of machine learning on the topic "What Kinds of Functions Do Neural Networks Learn?".
Abstract: Neural nets have made an amazing comeback during the past decade. Their empirical success has been truly phenomenal, but neural nets are poorly understood in a mathematical sense compared to classical methods like splines, kernels, and wavelets. This talk describes recent steps towards a mathematical theory of neural networks comparable to the foundations we have for classical nonparametric methods. Surprisingly, neural nets are minimax optimal in a wide variety of classical univariate function spaces, including those handled by splines and wavelets. In multivariate settings, neural nets are solutions to data-fitting problems cast in entirely new types of multivariate function spaces characterized through total variation (TV) measured in the Radon transform domain. Deep multilayer neural nets naturally represent compositions of functions in these Radon-TV spaces. This theory provides novel explanations for many notable empirical discoveries in deep learning and suggests new approaches to training neural networks.
This is joint work with Rahul Parhi.
Видео Robert Nowak - What Kinds of Functions Do Neural Networks Learn? канала One world theoretical machine learning
Abstract: Neural nets have made an amazing comeback during the past decade. Their empirical success has been truly phenomenal, but neural nets are poorly understood in a mathematical sense compared to classical methods like splines, kernels, and wavelets. This talk describes recent steps towards a mathematical theory of neural networks comparable to the foundations we have for classical nonparametric methods. Surprisingly, neural nets are minimax optimal in a wide variety of classical univariate function spaces, including those handled by splines and wavelets. In multivariate settings, neural nets are solutions to data-fitting problems cast in entirely new types of multivariate function spaces characterized through total variation (TV) measured in the Radon transform domain. Deep multilayer neural nets naturally represent compositions of functions in these Radon-TV spaces. This theory provides novel explanations for many notable empirical discoveries in deep learning and suggests new approaches to training neural networks.
This is joint work with Rahul Parhi.
Видео Robert Nowak - What Kinds of Functions Do Neural Networks Learn? канала One world theoretical machine learning
Показать
Комментарии отсутствуют
Информация о видео
14 октября 2021 г. 15:11:51
00:55:44
Другие видео канала
Lukasz Szpruch - Mean-Field Neural ODEs, Relaxed Control and Generalization Errors
Matthew Colbrook - Smale’s 18th Problem and the Barriers of Deep Learning
Micah Goldblum - Bridging the Gap between Deep Learning Theory and Practice
Yu Bai - How Important is the Train-Validation Split in Meta-Learning?
Anna Korba - Kernel Stein Discrepancy Descent
Anirbit Mukherjee - Provable Training of Neural Nets With One Layer of Activation
Kevin Miller - Ensuring Exploration and Exploitation in Graph-Based Active Learning
Theo Bourdais - Computational Hypergraph Discovery, a Gaussian Process framework
Yaoqing Yang - Predicting & improving generalization by measuring loss landscapes & weight matrices
Konstantinos Spiliopoulos - Mean field limits of neural networks: typical behavior and fluctuations
Nadia Drenska - A PDE Interpretation of Prediction with Expert Advice
Matthias Ehrhardt - Bilevel Learning for Inverse Problems
Peter Richtarik - The Resolution of a Question Related to Local Training in Federated Learning
Marcus Hutter - Testing Independence of Exchangeable Random Variables
Yury Korolev - Approximation properties of two-layer neural networks with values in a Banach space
Sophie Langer - Circumventing the curse of dimensionality with deep neural networks
Stephan Mandt - Compressing Variational Bayes: From neural data compression to video prediction
Derek Driggs - Barriers to Deploying Deep Learning Models During the COVID-19 Pandemic
Gal Vardi - Implications of the implicit bias in neural networks
Ziwei Ji - The dual of the margin: improved analyses and rates for gradient descent’s implicit bias
Qi Lei - Predicting What You Already Know Helps: Provable Self-Supervised Learning