Building (Better) Data Pipelines with Apache Airflow

AD:
Level-up on the skills most in-demand in 2021. Attend QCon Plus (May 17-28): http://bit.ly/3pfdF6I

If you are a senior software engineer, architect, or team lead and want to take your technical learning and personal development to a whole new level this year, join us at QCon Plus (virtual conference on May 17-28) and discover trends, best practices, and solutions leveraged by the world's most innovative software organizations.

Save your spot now: http://bit.ly/3pfdF6I

--------------------------------------------------------------------------------------------------------------------------------------
In this session, Sid Anand talks about Apache Airflow, an up-and-coming platform to programmatically author, schedule, manage, and monitor workflows. Airflow’s design requires users to define DAGs (directed acyclic graphs) a.k.a. workflows in Python code, so that DAGs can be managed via the same software engineering principles and practices used to manage any other code.

With more than 7600 GitHub stars, 2400 forks, 430 contributors, 150 companies officially using it, and 4600 commits, Apache Airflow is quickly gaining traction among data science, ETL engineering, data engineering, and DevOps communities at large.

Sid Anand currently serves as PayPal's Chief Data Engineer, focusing on ways to realize the value of data.

This video was recorded at QCon.ai 2018: https://bit.ly/2piRtLl

More videos from QCon.ai 2018 on InfoQ: https://bit.ly/2rNAT8z

The InfoQ Architects' Newsletter is your monthly guide to all the topics, technologies and techniques that every professional or aspiring software architect needs to know about. Over 200,000 software architects, team leads, CTOs are subscribed to it. Sign up here: https://bit.ly/2KqYfrs

Видео Building (Better) Data Pipelines with Apache Airflow канала InfoQ