Airflow XCom for Beginners - All you have to know in 10 mins
Airflow XCom for Beginners - All you have to know in 10 mins to share data between tasks.
👍 Smash the like button to become better at Airflow
❤️ Subscribe to my channel to become a master of Airflow
🏆 Take my course : https://www.udemy.com/course/the-ultimate-hands-on-course-to-master-apache-airflow/?couponCode=WEBSITE-13 to join the legends of Airflow
🚨 My Patreon: https://www.patreon.com/marclamberti to support my work and be friend for life
The materials: https://www.notion.so/Airflow-XCOM-All-you-have-to-know-in-10-mins-b6a937b449134bf3a9bbe44d5a758662
The blogs post: https://marclamberti.com/blog/airflow-xcom/
1. Use case
You have a data pipelines with 5 different tasks. The first task downloads data, 3 tasks train machine learning models, the last task chooses the best model. Each task training a machine learning model, produces an accuracy. According to this accuracy, you want to choose the best model in the last task. Question, how can you share the accuracies produced in the 3 training model tasks with the last task? XComs!
2. What is a XCom?
XCom stands for cross communication and allows to share messages and small amount of data between tasks in data pipelines. A XCom is composed of a key (identifier), a value (must be serializable), a timestamp (when was created), an execution date (to which dagrun the XCom belongs with), a task id (which task created the XCom), a dag id (same as the task but for the dag). XComs are stored in the database of Airflow.
3. How to push a XCom
2 ways, with the return keywords or with xcom_push. Any value returned from an operator is automatically pushed as a XCom with the key return_value. With xcom_push you have to specify the key as well as the value. To use xcom_push, you have to access the task instance object corresponding to your task.
4. How to pull a XCom
With xcom_pull. Again, to call xcom_pull, you have to access the task instance object of your task. xcom_pull expects two arguments, the key and a list of task ids.
5. XCom limitations
Different xcom size limit according to the database used.
Create implicit dependencies between your tasks.
Enjoy!
Видео Airflow XCom for Beginners - All you have to know in 10 mins канала Marc Lamberti
👍 Smash the like button to become better at Airflow
❤️ Subscribe to my channel to become a master of Airflow
🏆 Take my course : https://www.udemy.com/course/the-ultimate-hands-on-course-to-master-apache-airflow/?couponCode=WEBSITE-13 to join the legends of Airflow
🚨 My Patreon: https://www.patreon.com/marclamberti to support my work and be friend for life
The materials: https://www.notion.so/Airflow-XCOM-All-you-have-to-know-in-10-mins-b6a937b449134bf3a9bbe44d5a758662
The blogs post: https://marclamberti.com/blog/airflow-xcom/
1. Use case
You have a data pipelines with 5 different tasks. The first task downloads data, 3 tasks train machine learning models, the last task chooses the best model. Each task training a machine learning model, produces an accuracy. According to this accuracy, you want to choose the best model in the last task. Question, how can you share the accuracies produced in the 3 training model tasks with the last task? XComs!
2. What is a XCom?
XCom stands for cross communication and allows to share messages and small amount of data between tasks in data pipelines. A XCom is composed of a key (identifier), a value (must be serializable), a timestamp (when was created), an execution date (to which dagrun the XCom belongs with), a task id (which task created the XCom), a dag id (same as the task but for the dag). XComs are stored in the database of Airflow.
3. How to push a XCom
2 ways, with the return keywords or with xcom_push. Any value returned from an operator is automatically pushed as a XCom with the key return_value. With xcom_push you have to specify the key as well as the value. To use xcom_push, you have to access the task instance object corresponding to your task.
4. How to pull a XCom
With xcom_pull. Again, to call xcom_pull, you have to access the task instance object of your task. xcom_pull expects two arguments, the key and a list of task ids.
5. XCom limitations
Different xcom size limit according to the database used.
Create implicit dependencies between your tasks.
Enjoy!
Видео Airflow XCom for Beginners - All you have to know in 10 mins канала Marc Lamberti
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
Airflow DockerOperator: The Basics (and more 🤫)The Airflow BranchPythonOperator for Beginners in 10 minsBig data in Airflow? Discover the XCom Backends with AWS S3!Running Airflow 2.0 with Docker in 5 minsAirflow Sensors : Get started in 10 mins[Getting started with Airflow - 4] Passing metadata in between tasks with XCOMAirflow Dynamic DAGs: The powerful way with Jinja and YAMLPython HOW TO structure a Beginner OR Advanced Projects ?Airflow DAG: Coding your first DAG for BeginnersAirflow tutorial 7: Airflow variablesAirflow for Beginners - Run Spotify ETL Job in 15 minutes!Airflow tutorial 1: Introduction to Apache AirflowDon't Use Apache AirflowApache Airflow: Adios SubDAGs! Welcome TaskGroups!Airflow Helm Chart : Quick Start For Beginners in 10minsAirflow 2.0: What to expect?The TriggerDagRunOperator in Airflow! Create DAG dependencies at easePython Operator y Xcoms| Airflow 2.0 video 3Airflow tutorial 6: Build a data pipeline using Google Bigquery