AZURE DATA ENGINEERING PROJECT

Azure End-to-End Data Pipeline Project
This project demonstrates a complete cloud-native data pipeline built using Microsoft Azure services. It follows the modern Medallion Architecture (Bronze → Silver → Gold) and showcases how raw external data can be ingested, transformed, queried, and visualized — all on the cloud.

📌 Project Highlights
Data Ingestion using Azure Data Factory from external HTTP sources (e.g., GitHub)
Data Storage in Azure Data Lake Storage Gen2 with Bronze, Silver, and Gold layers
Data Transformation using Azure Databricks + PySpark
Data Modeling and SQL querying with Azure Synapse Analytics
Business Intelligence and Dashboards via Power BI
📽️ Project Demo
🔗 Watch the screen recording demo on LinkedIn:
▶ Video Post Link

📥 Embedded Video:
✍️ Article Walkthrough
📄 Read the full technical article:
📚 LinkedIn Article Post

📰 Embedded Article View:
🧱 Architecture Overview
[HTTP Source (GitHub)]
↓
[Azure Data Factory (ADF)]
↓
[Bronze Layer - Raw Data Storage]
↓
[Azure Databricks - Data Transformation (PySpark)]
↓
[Silver Layer - Cleaned Data Storage]
↓
[Azure Synapse Analytics - Querying & Modeling]
↓
[Gold Layer - Business Ready Data]
↓
[Power BI - Dashboards and Insights]
💡 Technologies Used
Azure Data Factory
Azure Data Lake Storage Gen2
Azure Databricks (PySpark)
Azure Synapse Analytics
Power BI
GitHub (as HTTP data source)
Entra ID (for secure access configuration)
📂 Repository Structure
├── adf/ # Azure Data Factory pipelines and configurations
│ ├── pipelines/ # Data ingestion pipeline definitions
│ └── linked_services/ # Connection configurations
│
├── databricks/ # Databricks notebooks
│ ├── bronze_to_silver/ # Transformation scripts for raw to cleaned data
│ └── silver_to_gold/ # Transformation scripts for analytics-ready data
│
├── synapse/ # Synapse Analytics SQL scripts
│ ├── views/ # SQL views for data modeling
│ └── stored_procedures/ # Stored procedures for data operations
│
├── powerbi/ # Power BI report templates
│ └── dashboards/ # Business intelligence dashboard designs
│
└── docs/ # Project documentation
├── architecture.md # Detailed architecture explanation
└── setup_guide.md # Deployment and configuration

Видео AZURE DATA ENGINEERING PROJECT канала Vishnu Murthy