Peter Richtarik - The Resolution of a Question Related to Local Training in Federated Learning
Presentation given by Peter Richtarik on 5th October 2022 in the one world seminar on the mathematics of machine learning on the topic "On the Resolution of a Theoretical Question Related to the Nature of Local Training in Federated Learning".
Abstract: We study distributed optimization methods based on the local training (LT) paradigm - achieving improved communication efficiency by performing richer local gradient-based training on the clients before parameter averaging - which is of key importance in federated learning. Looking back at the progress of the field in the last decade, we identify 5 generations of LT methods: 1) heuristic, 2) homogeneous, 3) sublinear, 4) linear, and 5) accelerated. The 5th generation, initiated by the ProxSkip method of Mishchenko et al (2022) and its analysis, is characterized by the first theoretical confirmation that LT is a communication acceleration mechanism. In this talk, I will explain the problem, its solution, and some subsequent work generalizing, extending and improving the ProxSkip method in various ways.
References:
1. Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich and Peter Richtárik. ProxSkip: Yes! Local gradient steps provably lead to communication acceleration! Finally! Proceedings of the 39th International Conference on Machine Learning, 2022
2. Grigory Malinovsky, Kai Yi and Peter Richtárik. Variance reduced ProxSkip: Algorithm, theory and application to federated learning, arXiv:2207.04338, 2022
3. Laurent Condat and Peter Richtárik. RandProx: Primal-dual optimization algorithms with randomized proximal updates, arXiv:2207.12891, 2022
4. Abdurakhmon Sadiev, Dmitry Kovalev and Peter Richtárik. Communication acceleration of local gradient methods via an accelerated primal-dual algorithm with inexact prox, arXiv:2207.03957, 2022
Видео Peter Richtarik - The Resolution of a Question Related to Local Training in Federated Learning канала One world theoretical machine learning
Abstract: We study distributed optimization methods based on the local training (LT) paradigm - achieving improved communication efficiency by performing richer local gradient-based training on the clients before parameter averaging - which is of key importance in federated learning. Looking back at the progress of the field in the last decade, we identify 5 generations of LT methods: 1) heuristic, 2) homogeneous, 3) sublinear, 4) linear, and 5) accelerated. The 5th generation, initiated by the ProxSkip method of Mishchenko et al (2022) and its analysis, is characterized by the first theoretical confirmation that LT is a communication acceleration mechanism. In this talk, I will explain the problem, its solution, and some subsequent work generalizing, extending and improving the ProxSkip method in various ways.
References:
1. Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich and Peter Richtárik. ProxSkip: Yes! Local gradient steps provably lead to communication acceleration! Finally! Proceedings of the 39th International Conference on Machine Learning, 2022
2. Grigory Malinovsky, Kai Yi and Peter Richtárik. Variance reduced ProxSkip: Algorithm, theory and application to federated learning, arXiv:2207.04338, 2022
3. Laurent Condat and Peter Richtárik. RandProx: Primal-dual optimization algorithms with randomized proximal updates, arXiv:2207.12891, 2022
4. Abdurakhmon Sadiev, Dmitry Kovalev and Peter Richtárik. Communication acceleration of local gradient methods via an accelerated primal-dual algorithm with inexact prox, arXiv:2207.03957, 2022
Видео Peter Richtarik - The Resolution of a Question Related to Local Training in Federated Learning канала One world theoretical machine learning
Показать
Комментарии отсутствуют
Информация о видео
6 октября 2022 г. 14:13:29
01:04:35
Другие видео канала
Lukasz Szpruch - Mean-Field Neural ODEs, Relaxed Control and Generalization Errors
Matthew Colbrook - Smale’s 18th Problem and the Barriers of Deep Learning
Yu Bai - How Important is the Train-Validation Split in Meta-Learning?
Anna Korba - Kernel Stein Discrepancy Descent
Anirbit Mukherjee - Provable Training of Neural Nets With One Layer of Activation
Kevin Miller - Ensuring Exploration and Exploitation in Graph-Based Active Learning
Theo Bourdais - Computational Hypergraph Discovery, a Gaussian Process framework
Yaoqing Yang - Predicting & improving generalization by measuring loss landscapes & weight matrices
Konstantinos Spiliopoulos - Mean field limits of neural networks: typical behavior and fluctuations
Nadia Drenska - A PDE Interpretation of Prediction with Expert Advice
Matthias Ehrhardt - Bilevel Learning for Inverse Problems
Marcus Hutter - Testing Independence of Exchangeable Random Variables
Yury Korolev - Approximation properties of two-layer neural networks with values in a Banach space
Sophie Langer - Circumventing the curse of dimensionality with deep neural networks
Stephan Mandt - Compressing Variational Bayes: From neural data compression to video prediction
Derek Driggs - Barriers to Deploying Deep Learning Models During the COVID-19 Pandemic
Gal Vardi - Implications of the implicit bias in neural networks
Ziwei Ji - The dual of the margin: improved analyses and rates for gradient descent’s implicit bias
Qi Lei - Predicting What You Already Know Helps: Provable Self-Supervised Learning
Alessandro Scagliotti - Deep Learning Approximation of Diffeomorphisms via Linear-Control Systems