Elegant data pipelining with Apache Airflow - Bolke de Bruin
PyData Amsterdam 2018
Batch data processing, historically known as ETL, is extremely challenging. It’s time-consuming, brittle, and often unrewarding. Not only that, it’s hard to operate, evolve, and troubleshoot. As ETL pipelines grow in complexity, and as data teams grow in numbers, using methodologies that provide clarity isn’t a luxury, it’s a necessity.
--
www.pydata.org
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.
PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases. 00:00 Welcome!
00:10 Help us add time stamps or captions to this video! See the description for details.
Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: https://github.com/numfocus/YouTubeVideoTimestamps
Видео Elegant data pipelining with Apache Airflow - Bolke de Bruin канала PyData
Batch data processing, historically known as ETL, is extremely challenging. It’s time-consuming, brittle, and often unrewarding. Not only that, it’s hard to operate, evolve, and troubleshoot. As ETL pipelines grow in complexity, and as data teams grow in numbers, using methodologies that provide clarity isn’t a luxury, it’s a necessity.
--
www.pydata.org
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.
PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases. 00:00 Welcome!
00:10 Help us add time stamps or captions to this video! See the description for details.
Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: https://github.com/numfocus/YouTubeVideoTimestamps
Видео Elegant data pipelining with Apache Airflow - Bolke de Bruin канала PyData
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
Future of Data EngineeringScalable Data Ingestion Architecture Using Airflow and Spark | Komodo HealthLearn Docker in 12 Minutes 🐳Airflow in Practice Stop Worrying Start Loving DAGs - Sarah SchattschneiderTalk: Daniel Imberman - Bridging Data Science and Data Infrastructure with Apache AirflowHow to Become a Data EngineerFIT INTERVIEW EXAMPLE WITH FORMER MCKINSEY INTERVIEWERRunning Apache Airflow Reliably with Kubernetes | AstronomerEddie Bell: Weak supervision: a new paradigm for unreliable data | PyData London 2019"Dynamic Data Pipelining with Luigi" - Trey Hakanson (Pyohio 2019)Michał Karzyński - Developing elegant workflows in Python code with Apache AirflowDesigning ETL Pipelines with Structured Streaming and Delta Lake— How to Architect Things RightAccelerating Data Science with RAPIDS - Keith KrausData Lineage with Apache Airflow | DatakinIodide and Pyodide: Bringing Data Science Computation to the Web Browser - Michael DroettboomFunctional Programming in ScalaJob Execution Systems: What is the difference between Jenkins, Rundeck, Airflow, Gitlab CI and otherFrom Idea to Model: Productionizing Data Pipelines with Apache AirflowBuilding reuseable and trustworthy ELT pipelines (A templated approach)Delta Lake: Reliability and Data Quality for Data Lakes and Apache Spark by Michael Armbrust