How to submit Spark jobs to EMR cluster from Airflow
In this video we go over the steps on how to create a temporary EMR cluster, submit jobs to it, wait for the jobs to complete and terminate the cluster, the Airflow-way.
Blog: https://www.startdataengineering.com/post/how-to-submit-spark-jobs-to-emr-cluster-from-airflow/
Видео How to submit Spark jobs to EMR cluster from Airflow канала StartDataEngineering
Blog: https://www.startdataengineering.com/post/how-to-submit-spark-jobs-to-emr-cluster-from-airflow/
Видео How to submit Spark jobs to EMR cluster from Airflow канала StartDataEngineering
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
how to setup dbt cloud with snowflake
how to unit test sql transforms in dbt
How to Backfill a SQL query using Apache Airflow
what are Common Table Expressions(CTEs)?
SQL window functions tutorial
How to Pull Data from an API, Using AWS Lambda
data engineering project - stream edition
ETL vs ELT
what is staging ?
what is data warehousing
great expectations tutorial
Concepts to build an ELT pipeline
3 Key techniques, to optimize your Apache Spark code
Why, When and How to use Apache Kafka
Beginner Data Engineering Project
How to do CDC using debezium, kafka and postgres
dbt(data build tool) tutorial