Top 10 Data Engineering Mistakes

A large fraction of big data projects fail to deliver return of investment, or take years before they do so. The reasons are typically a combination of project management, leadership, organisation, available competence, and technical failures. In this presentation, I will focus on the technical aspects, and present the most common or costly data engineering mistakes that I have experienced when building scalable data processing technology over the last five years, as well as advice for how to avoid them. The presentation includes war stories from large scale production environments, some that lead to reprocessing of petabytes of data, or DDoSing critical services with a Hadoop cluster, and what we learnt from the incidents.

EVENT:

#bbuzz 2018

SPEAKER:

Lars Albertsson

PERMISSIONS:

Original video was published with the Creative Commons Attribution license (reuse allowed).

CREDITS:

Original video source: https://www.youtube.com/watch?v=mv7PLnwzLpM&t=132s

Видео Top 10 Data Engineering Mistakes канала Coding Tech