No one at Google uses MapReduce anymore - Cloud Dataflow explained for dummies
Warning: this an an algorithmics talk, and it also involves parallel processing. The MapReduce paper, published by Google 10 years ago (2004!), sparked the parallel processing revolution and gave birth to countless open source and research projects. We have been busy since then and the MapReduce model is now officially obsolete. The new data processing models we use are called Flume (for the processing pipeline definition) and MillWheel for the real-time dataflow orchestration. We are releasing them as a public tool called Cloud Dataflow which allows you to specify both batch and real-time data processing pipelines and have them deployed and maintained automatically - and yes, dataflow can deploy *lots* of machines to handle Google-scale problems. What is the magic behind the scenes ? What is the post-MapReduce dataflow model ? What are the flow optimisation algorithms ? Read the papers or come for a walk through the algorithms with me.
Authors:
Martin Gorner
Martin is passionate about science, technology, coding, algorithms and everything in between. He graduated from Mines Paris Tech, enjoyed his first engineering years in the computer architecture group of ST Microlectronics and then spent the next 11 years shaping the nascent eBook market, starting with the Mobipocket startup, which later became the software part of the Amazon Kindle and its mobile variants. He joined Google Developer Relations in 2011 and now focuses on entrepreneurship outreach.
Blog: https://plus.google.com/+MartinGorner
Thomas Park
Google software engineer working to put the power of BigQuery and Dremel in the hands of developers worldwide.
Blog: http://googledevelopers.blogspot.com/
Видео No one at Google uses MapReduce anymore - Cloud Dataflow explained for dummies канала Parleys
Authors:
Martin Gorner
Martin is passionate about science, technology, coding, algorithms and everything in between. He graduated from Mines Paris Tech, enjoyed his first engineering years in the computer architecture group of ST Microlectronics and then spent the next 11 years shaping the nascent eBook market, starting with the Mobipocket startup, which later became the software part of the Amazon Kindle and its mobile variants. He joined Google Developer Relations in 2011 and now focuses on entrepreneurship outreach.
Blog: https://plus.google.com/+MartinGorner
Thomas Park
Google software engineer working to put the power of BigQuery and Dremel in the hands of developers worldwide.
Blog: http://googledevelopers.blogspot.com/
Видео No one at Google uses MapReduce anymore - Cloud Dataflow explained for dummies канала Parleys
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
![What is Dataflow?](https://i.ytimg.com/vi/KalJ0VuEM7s/default.jpg)
![What is MapReduce?](https://i.ytimg.com/vi/43fqzaSH0CQ/default.jpg)
![What is Hadoop?: SQL Comparison](https://i.ytimg.com/vi/MfF750YVDxM/default.jpg)
![How Target leverages Google Cloud](https://i.ytimg.com/vi/djQIFqY5cJ4/default.jpg)
![Natural Language Generation at Google Research](https://i.ytimg.com/vi/MNvT5JekDpg/default.jpg)
![Webinar: Building a real-time analytics pipeline with BigQuery and Cloud Dataflow (EMEA)](https://i.ytimg.com/vi/kdmAiQeYGgE/default.jpg)
![Cloud Build - Create a CI/CD Pipeline](https://i.ytimg.com/vi/Zd014DjonqE/default.jpg)
![Python: Lambda, Map, Filter, Reduce Functions](https://i.ytimg.com/vi/cKlnR-CB3tk/default.jpg)
![Getting a Job at Google: The Secrets Nobody Tells You](https://i.ytimg.com/vi/YVo0vkiagc0/default.jpg)
![Google Team Match](https://i.ytimg.com/vi/fG3noON-IWo/default.jpg)
![Interview with Nicolas from Aldebaran](https://i.ytimg.com/vi/mA8EqzNHgeQ/default.jpg)
![30 Jenkins features and plugins you wished you had known about before! by Joep Weijers](https://i.ytimg.com/vi/6BIry0cepz4/default.jpg)
![Cloud Foundry - Quick Introduction | Tech Primers](https://i.ytimg.com/vi/vzSuYab2q5M/default.jpg)
![If you are going to San Francisco](https://i.ytimg.com/vi/kvimsPsOguY/default.jpg)
![Container management and deployment: from development to production (Google Cloud Next '17)](https://i.ytimg.com/vi/XL9CQobFB8I/default.jpg)
![Cloud Hosting vs Traditional Hosting](https://i.ytimg.com/vi/4lLskYgpKKo/default.jpg)
![MongoDB Tutorial - Modeling with MongoDB](https://i.ytimg.com/vi/4rhKKFbbYT4/default.jpg)
![Cloud Pub/Sub Overview - ep. 1](https://i.ytimg.com/vi/cvu53CnZmGI/default.jpg)
![Learn MapReduce with Playing Cards](https://i.ytimg.com/vi/bcjSe0xCHbE/default.jpg)
![Networking with Kubernetes](https://i.ytimg.com/vi/WwQ62OyCNz4/default.jpg)