Day 12 Part 5: Medallion Architecture Load to Bronze Layer Azure Databricks for Data Engineers

This video will explain how to ingest load to Bronze Layer.

Don't miss out on this opportunity to excel!
🚀 **Course:** Master Azure Data Engineering
📅 **Last Date:** 15 Jan 2025

Course Registration: https://tinyurl.com/5n7aatdm

Don't miss out on this opportunity to upscale your skills and dive deep into the realm of data engineering! Reserve your spot now! 🎉

#hiring #career #databricks #azure #career #hiring #databricksanalytics

Don't miss out on this opportunity to upscale your skills and dive deep into the realm of data engineering! Reserve your spot now! 🎉

#hiring #career #databricks #azure #career #hiring #databricksanalytics

Typically, we will have three layers bronze, silver, and gold.
So let's understand this architecture end to end, starting from the data sources. So the data sources can be Kafka streaming data lakes which we will generally use it.
Or it can be databases where your data is generally getting ingested. All these will be where the data is coming from.
You can use any of the ETL tool to get this data ingested from an external system. So, this generally sits in a data lake folder which we will use in our project. So being specific with Azure we have an Azure Data Factory which is an ETL tool to ingest the data.

Bronze Layer: Now there comes some medallion architecture and that starts with the bronze layer.
So this bronze layer typically also called as a raw layer. In this the data is first ingested into the system, as is usually in the bronze layer. The data will be loaded incrementally and this will grow in time. The ingested data into the bronze layer can be a combination of batch and streaming, although the data that is kept here is mostly raw.

So this is the primary zone where we will have the exact same data that we receive from our sources without having any modification.
Step 1: Create Storage account and directories
Step 2: Create Azure Databricks Resource
Step 3: Access Connector for Azure Databricks
Step 4: Add role assignment in Access Control (IAM)
Step 5: Create Metastore from Databricks Notebook
Step 6: create a dev-catalog
Step 7: Create a Compute Resource
Step 8: Create External Storages
Step 9: Part 0 Medallion Architecture Project overview
Step 10: Part 1 Project set up Creating bronze Tables Dynamically
Step 11: Load to Bronze
Step 12: Silver Traffic Transformations
Step 13: To re-use common functions and variables
Step 14: Silver - Roads Transformation

💡𝐀𝐫𝐞 𝐲𝐨𝐮 𝐩𝐥𝐚𝐧𝐧𝐢𝐧𝐠 𝐭𝐨 𝐥𝐞𝐚𝐫𝐧 𝐀𝐳𝐮𝐫𝐞 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠 𝐰𝐢𝐭𝐡 𝐇𝐚𝐧𝐝𝐬-𝐨𝐧 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐞?
🚀 𝐖𝐚𝐧𝐭 𝐭𝐨 𝐜𝐥𝐚𝐫𝐢𝐟𝐲 𝐚𝐧𝐲 𝐜𝐞𝐫𝐭𝐢𝐟𝐢𝐜𝐚𝐭𝐞?

🌟 This playlist might be helpful for in terms of learning with hands-on demo: Master key commands and techniques for real-world implementations.

✅ Day 11: What is Autoloader - Schema inference: Hands-on Azure Databricks for Data Engineers:
🔄 https://youtu.be/6U7Lb6FOQVU
✅ Day 10: What is Spark Structured Streaming – learn basics: Azure Databricks for Data Engineers
🔄 https://youtu.be/8aSy-tqtgvY
✅ Day 9: What is Unity Catalog: Managed and External Tables in Unity Catalog Azure Databricks
🔄 https://youtu.be/LPxJbvj2mqc

✅ Day 8: Optimize Upsert command in DataBricks
🔄 https://youtu.be/741tXZdwz-E

✅ Day 7: What is Schema Evolution and Vacuum command in Azure Databricks for Data Engineers
🔄 https://youtu.be/P6xaEZaciOU?si=f-UI9I1JuAZaO65d

✅ Day 6: Create Delta Lake and Overwrite drawbacks of ADLS: Master Azure Databricks for Data Engineers Day 6
🔄 https://youtu.be/hZEwI39iwTY?si=Etc3_cWNx1Yl0SL9

✅ Day 5: DBUtils Notebook in Databricks- Master Azure Databricks for Data Engineers Day 5
🔄 https://youtu.be/-EehWASevEA?si=VW3vKFPeDKYHhA_J

✅ Day 4: Widget Utilities in Databricks- Master Azure Databricks for Data Engineers Day 4
🔄 https://youtu.be/vxACLPERakc?si=_pr_PmE4GFrwUWFH

✅ Day 3: Magic commands and DBUtils in Databricks -Master Azure Databricks for Data Engineers Day 3
🔄 https://youtu.be/ZpszcjwISKU?si=_sVKa7iVViXKCR7Z

✅ Day 2: Understanding notebook and Markdown basics - Master Azure Databricks for Data Engineers Day 2
🔄 https://youtu.be/MTHASycNtoo?si=fjb-5qPZQKOD_9Hw

✅ Day 1: Databricks - Master Azure Databricks for Data Engineers Day 1
🔄 https://youtu.be/8M5GroxqCRI?si=jMiOJ1BPmEvgtIJM

⚙️ Complete DataBricks Azure Play List Link:
https://youtube.com/playlist?list=PLmhdUvQQZFDnvKnPr2duoGyL21PjaQLW9&si=es1g64jBXYiqyDTn

🔥 Key Highlights:
Learn step-by-step how to work with Azure Data Lake Storage (ADLS) and Delta Lake in Databricks.
Understand file formats like Parquet and Delta, schema evolution, and versioning.
🎯 **Don’t Forget to**:
👍 Like if this guide helps!
🔁 Share with your network!
📥 Save for future reference!
💬 Comment your insights or questions below!

♻️ Follow Dr Sachin Saxena for more Data Engineering related posts !!!!
LinkedIn Profile of author:
https://www.linkedin.com/in/sachin-saxena-graphic-designer/
For any Query mail me at: sachinsax@gmail.com

Видео Day 12 Part 5: Medallion Architecture Load to Bronze Layer Azure Databricks for Data Engineers канала Code with Kristi