Data pipelines from zero to solid
(Recorded at Jfokus 2016. http://www.jfokus.com)
Data pipelines from zero to solid
Collect lots of data, do something mysterious, make profit! The big data buzz affects everyone. We all have data, and we all have a hunch that we may be able to get something valuable out of it, but the step in between is somewhat foggy, both in terms of engineering and science.- This presentation is an attempt do demystify the practice of building reliable data processing pipelines. We will go through the necessary pieces needed to build a stable processing platform: data ingestion, processing engines, workflow management, schemas, and pipeline development processes. The presentation will also include component choice considerations and recommendations, as well as best practices and pitfalls to avoid, most learnt through expensive mistakes.
Lars Albertsson
Lars Albertsson has worked with data-intensive and scalable applications at Google, Spotify, Schibsted Media Group, natural language processing startup Recorded Future, and with stock exchange systems. He is now an independent consultant, helping companies build scalable data processing solutions.
Видео Data pipelines from zero to solid канала Jfokus
Data pipelines from zero to solid
Collect lots of data, do something mysterious, make profit! The big data buzz affects everyone. We all have data, and we all have a hunch that we may be able to get something valuable out of it, but the step in between is somewhat foggy, both in terms of engineering and science.- This presentation is an attempt do demystify the practice of building reliable data processing pipelines. We will go through the necessary pieces needed to build a stable processing platform: data ingestion, processing engines, workflow management, schemas, and pipeline development processes. The presentation will also include component choice considerations and recommendations, as well as best practices and pitfalls to avoid, most learnt through expensive mistakes.
Lars Albertsson
Lars Albertsson has worked with data-intensive and scalable applications at Google, Spotify, Schibsted Media Group, natural language processing startup Recorded Future, and with stock exchange systems. He is now an independent consultant, helping companies build scalable data processing solutions.
Видео Data pipelines from zero to solid канала Jfokus
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
Socio-Technical Smells: How Technical Problems Cause Organizational Friction by Adam TornhillMoving to Java 17 in production by Andrzej GrzesikGetting up to speed with Kafka Connect by Kate StanleyTyped JavaScript with TypeScript, Flow and ElmSami Ekblad @ Jfokus 2010TDD: seriously, try it! - Nacho Cougil (Jfokus Brown Bag series)Patterns for building resilient software systems by Adrian HornsbyStreamlining large-scale Java development using Error Prone by Sander MakGame over or game changing? Why software development may never be the same again by Steve Poole55 New Features in JDK 9 by Simon RitterCompliance-as-Code: How to do regulated business using public cloud by Emelie OhlsonLet's move to X - Comparing the security of architecture patterns by Sebastian OlssonDistributed Deep Learning by Jim DowlingDon't bore your cores!Deploying 5 Java Frameworks in 5 Different Ways to the Cloud by Ed BurnsStaying Human While Coding by April WenselAutomated Tests Are Not Enough by Henrik WarneJfokus 2017 mood videoLearning from your bugs by Henrik WarneMoving Java Forward Faster by Mark ReinholdThe Heart of Domain-Driven Design - Andrew Harmel-Law (Jfokus Brown Bag series)