Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
In this video from 2018 Swiss HPC Conference, Torsten Hoefler from (ETH) Zürich presents: Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis.
"Deep Neural Networks (DNNs) are becoming an important tool in modern computing applications. Accelerating their training is a major challenge and techniques range from distributed algorithms to low-level circuit design. In this talk, we describe the problem from a theoretical perspective, followed by approaches for its parallelization.
Specifically, we present trends in DNN architectures and the resulting implications on parallelization strategies. We discuss the different types of concurrency in DNNs; synchronous and asynchronous stochastic gradient descent; distributed system architectures; communication schemes; and performance modeling. Based on these approaches, we extrapolate potential directions for parallelism in deep learning."
Learn more: http://hpcadvisorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Видео Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis канала InsideHPC Report
"Deep Neural Networks (DNNs) are becoming an important tool in modern computing applications. Accelerating their training is a major challenge and techniques range from distributed algorithms to low-level circuit design. In this talk, we describe the problem from a theoretical perspective, followed by approaches for its parallelization.
Specifically, we present trends in DNN architectures and the resulting implications on parallelization strategies. We discuss the different types of concurrency in DNNs; synchronous and asynchronous stochastic gradient descent; distributed system architectures; communication schemes; and performance modeling. Based on these approaches, we extrapolate potential directions for parallelism in deep learning."
Learn more: http://hpcadvisorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Видео Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis канала InsideHPC Report
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
Distributed TensorFlow training (Google I/O '18)
Hogwild for Machine Learning on Multicore
Geoffrey Hinton talk "What is wrong with convolutional neural nets ?"
Study Techniques - The Good, Bad, & Useless
Parallel Streams, CompletableFuture, and All That: Concurrency in Java 8
Distributed Systems in One Lesson by Tim Berglund
PyTorch Lightning - William Falcon
An Introduction to GPU Programming with CUDA
The Issue with Machine Learning in Finance
Ian Goodfellow: Adversarial Machine Learning (ICLR 2019 invited talk)
Distributed Training in the Cloud
SysML 19: Jia Zhihao, Beyond Data and Model Parallelism for Deep Neural Networks![Lecture 17.6 — Large Scale Machine Learning | Map Reduce And Data Parallelism — [ Andrew Ng ]](https://i.ytimg.com/vi/TCA2VuHTHcM/default.jpg)
Lecture 17.6 — Large Scale Machine Learning | Map Reduce And Data Parallelism — [ Andrew Ng ]
Training on multiple GPUs and multi-node training with PyTorch DistributedDataParallel
Machine Learning in R: Speed up Model Building with Parallel Computing![Lecture 17.2 — Large Scale Machine Learning | Stochastic Gradient Descent — [ Andrew Ng ]](https://i.ytimg.com/vi/W9iWNJNFzQI/default.jpg)
Lecture 17.2 — Large Scale Machine Learning | Stochastic Gradient Descent — [ Andrew Ng ]
Understanding Mini-Batch Gradient Dexcent (C2W2L02)
Distributed Machine Learning with Python![[deep learning] Federated Learning - training on decentralized data](https://i.ytimg.com/vi/KxZXhzfDgik/default.jpg)
[deep learning] Federated Learning - training on decentralized data
How to Manage GPU Resource Utilization in Tensorflow and Keras