Incremental Loading in Apache Airflow - Handle Large Data More Efficiently

Big data can be difficult data. Design your Airflow pipelines to process data more efficiently with incremental loading.

We'll see how to use template variables to process small chunks of our source data.

Chapters:
00:00 Introduction
00:25 What is Incremental Loading?
01:53 Airflow Data Interval
03:50 Example DAG: Donut App Orders
07:05 Building the DAG
10:18 Rendering Template Variables
11:49 Re-Running a DAG
14:53 Outro

Code:
https://github.com/kishanpatel789/kp_data_dev_blog_repos/tree/main/airflow_incremental_loading

Blog post:
https://kpdata.dev/blog/airflow-incremental-loading/

Other Resources
Airflow Docs: https://airflow.apache.org/docs/apache-airflow/stable/index.html
Airflow Template Variables: https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#variables

#airflow #dataengineering #python

Видео Incremental Loading in Apache Airflow - Handle Large Data More Efficiently канала KP Data Dev