Fabian Höring - Building reproducible distributed applications at scale
"Building reproducible distributed applications at scale
EuroPython 2020 - Talk - 2020-07-24 - Brian
Online
By Fabian Höring
Packaging in Python is hard. Packaging is particularly hard when code needs to run in a distributed computing environment where it is difficult to know what runs where and which parts of the code are available to run there.
In this talk we will present different ways to ship Python code to a compute cluster, what Python's ""pickling"" feature has to do with this, what self contained executables are and the challenges we met when shipping Python code to a cluster with 1000s of nodes running 1000s of jobs like TensorFlow or Spark.
As an example, we will show how one can run a PySpark job on top of S3 storage using PEX as a self contained executable artifact. Finally we will explain how those ideas generalize for different Jobs (like Tensorflow, Dask), different virtual environments (like Anaconda or vanilla Python virtual envs) and different distributed storage's (like S3 or HDFS).
The auditor will get an overview of the challenges of Python packaging for distributed applications and see code samples that can be applied in his own project.
License: This video is licensed under the CC BY-NC-SA 3.0 license: https://creativecommons.org/licenses/by-nc-sa/3.0/
Please see our speaker release agreement for details: https://ep2020.europython.eu/events/speaker-release-agreement/
"
Видео Fabian Höring - Building reproducible distributed applications at scale канала EuroPython Conference
EuroPython 2020 - Talk - 2020-07-24 - Brian
Online
By Fabian Höring
Packaging in Python is hard. Packaging is particularly hard when code needs to run in a distributed computing environment where it is difficult to know what runs where and which parts of the code are available to run there.
In this talk we will present different ways to ship Python code to a compute cluster, what Python's ""pickling"" feature has to do with this, what self contained executables are and the challenges we met when shipping Python code to a cluster with 1000s of nodes running 1000s of jobs like TensorFlow or Spark.
As an example, we will show how one can run a PySpark job on top of S3 storage using PEX as a self contained executable artifact. Finally we will explain how those ideas generalize for different Jobs (like Tensorflow, Dask), different virtual environments (like Anaconda or vanilla Python virtual envs) and different distributed storage's (like S3 or HDFS).
The auditor will get an overview of the challenges of Python packaging for distributed applications and see code samples that can be applied in his own project.
License: This video is licensed under the CC BY-NC-SA 3.0 license: https://creativecommons.org/licenses/by-nc-sa/3.0/
Please see our speaker release agreement for details: https://ep2020.europython.eu/events/speaker-release-agreement/
"
Видео Fabian Höring - Building reproducible distributed applications at scale канала EuroPython Conference
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
![Radoslav Georgiev - Django structure for scale and longevity](https://i.ytimg.com/vi/yG3ZdxBb1oo/default.jpg)
![Miloslav Pojman - HTTP/3 – Why should I care?](https://i.ytimg.com/vi/vXbP7EmpErE/default.jpg)
![Dougal Matthews - Effective Code Review](https://i.ytimg.com/vi/uIwl01Nazdg/default.jpg)
![Dan Taylor - Get Productive with Python in Visual Studio Code](https://i.ytimg.com/vi/6YLMWU-5H9o/default.jpg)
![Ching Lam Choi - Corona-Net](https://i.ytimg.com/vi/57gCQSUv2fY/default.jpg)
![David Beazley - Die Threads](https://i.ytimg.com/vi/xOyJiN3yGfU/default.jpg)
![Dustin Ingram - Static Typing in Python](https://i.ytimg.com/vi/cmmPDYa0gWo/default.jpg)
![Stefan Behnel - Get up to speed with Cython 3.0](https://i.ytimg.com/vi/lxh4lsvEBhI/default.jpg)
![Distributed Systems in One Lesson by Tim Berglund](https://i.ytimg.com/vi/Y6Ev8GIlbxc/default.jpg)
![T. Petkos, G. Zisopoulos - Ensuring data integrity with asynchronous programming in a cloud IoT core](https://i.ytimg.com/vi/NuBTiOwMv6o/default.jpg)
![EuroPython Conference Live Stream](https://i.ytimg.com/vi/IMzs7A6kxrk/default.jpg)
![Sebastian Witowski - Writing faster Python](https://i.ytimg.com/vi/YjHsOrOOSuI/default.jpg)
![Stefan Baerisch - Tests that (Almost) Write Themselves](https://i.ytimg.com/vi/CSwm95DLRf4/default.jpg)
![Patrick Arminio - GraphQL in Python 🍓](https://i.ytimg.com/vi/38uCu6ZEvfo/default.jpg)
![Jason McDonald - Writing Zenlike Python](https://i.ytimg.com/vi/ZHY5SCBckwg/default.jpg)
![Siddha Ganju - 30 Golden Rules of Deep Learning Performance](https://i.ytimg.com/vi/B98m2k7CKYc/default.jpg)
![EuroPython 2020 - Ni Track Stream - Part 2 - 2020-07-23](https://i.ytimg.com/vi/rMglI4StVLY/default.jpg)
![Miki Tebeka - IPython: The Productivity Booster](https://i.ytimg.com/vi/Zmo2ZN1SJ_Q/default.jpg)
![Reuven Lerner - How to sort anything](https://i.ytimg.com/vi/Z3c2LvEJeu0/default.jpg)
![Yury Selivanov - async/await in Python 3.5 and why it is awesome](https://i.ytimg.com/vi/m28fiN9y_r8/default.jpg)