Building Robust ETL Pipelines with Apache Spark - Xiao Li

Stable and robust ETL pipelines are a critical component of the data infrastructure of modern enterprises. ETL pipelines ingest data from a variety of sources and must handle incorrect, incomplete or inconsistent records and produce curated, consistent data for consumption by downstream applications.

In this talk, we'll take a deep dive into the technical details of how Apache Spark "reads" data and discuss how Spark 2.2's flexible APIs; support for a wide variety of datasources; state of art Tungsten execution engine; and the ability to provide diagnostic feedback to users, making it a robust framework for building end-to-end ETL pipelines.

Overview:
1) What’s an ETL Pipeline?

2) Using Spark SQL for ETL
- Extract: Dealing with Dirty Data (Bad Records or Files)
- Extract: Multi-line JSON/CSV Support
- Transformation: High-order functions in SQL
- Load: Unified write paths and interfaces

3) New Features in Spark 2.3
- Performance (Data Source API v2, Python UDF)

View slides:
https://www.slideshare.net/databricks/building-robust-etl-pipelines-with-apache-spark

Related articles:
Integrating Apache Airflow and Databricks: Building ETL pipelines with Apache Spark
https://databricks.com/blog/2016/12/08/integrating-apache-airflow-databricks-building-etl-pipelines-apache-spark.html

Writing Data Engineering Pipelines in Apache Spark on Databricks
https://databricks.com/blog/2016/09/06/writing-data-engineering-pipelines-in-apache-spark-on-databricks.html

About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: https://databricks.com/product/unified-data-analytics-platform

Connect with us:
Website: https://databricks.com
Facebook: https://www.facebook.com/databricksinc
Twitter: https://twitter.com/databricks
LinkedIn: https://www.linkedin.com/company/databricks
Instagram: https://www.instagram.com/databricksinc/

Видео Building Robust ETL Pipelines with Apache Spark - Xiao Li канала Databricks