Azure data pipeline project | Data Factory | ADLS Gen2 | Databricks | Part 1/3: Project Architecture

In this hands-on project, we will build a complete data solution using Microsoft Azure from scratch.

Move on-premises CSV data to the Azure cloud using a Self-hosted Integration Runtime
Set up a Logic App to receive automatic email notifications for pipeline success or failure
Perform thorough data cleaning and transformation in Azure Databricks
Store the final cleaned data in Azure Data Lake Storage Gen2

By the end of this project, you’ll understand how to design a reliable, automated, and scalable data pipeline in Azure.

Perfect for beginners and intermediate learners aiming to master real-world Azure Data Engineering workflows.

Dataset link:
https://www.kaggle.com/datasets/amruthayenikonda/dirty-dataset-to-practice-data-cleaning

Видео Azure data pipeline project | Data Factory | ADLS Gen2 | Databricks | Part 1/3: Project Architecture канала Nucleus - Data Engineering