Microsoft Fabric: Incremental ETL for Warehouse using Dataflow Gen 2, SQL Procedure, Data Pipeline
This video will discover the ways to run Incremental ETL(Data Load) using Dataflow Gen2, SQL Procedure, and Data Pipeline in Warehouse. Warehouse Support read and write SQL to SQL procedure has been used.
00:00 Overview of Incremental ETL
10:00 Create Warehouse and Load Data
26:00 What is the issue how to fix it
33:00 Load Data Using SQL procedure
36:30 Create Pipeline
30:00 Load Incremental Data
CREATE TABLE [dbo].[incr_info]
(
[tablename1] [varchar](4000) NULL,
[collectdate] [datetime2](6) NULL
)
insert into [dbo].[incr_info] values('sales', '2018-01-01');
CREATE TABLE dbo.sales_stg
(
order_no bigint,
item_id bigint,
sales_date datetime2(6),
customer_id bigint,
city_id bigint,
qty float,
price float,
cogs float,
discount_percent float
)
;
CREATE TABLE dbo.sales
(
order_no bigint,
item_id bigint,
sales_date datetime2(6),
customer_id bigint,
city_id bigint,
qty float,
price float,
cogs float,
discount_percent float
)
;
CREATE PROCEDURE dbo.loadSales
AS
BEGIN
insert into dbo.sales(
order_no ,
item_id ,
sales_date ,
customer_id ,
city_id ,
qty ,
price ,
cogs ,
discount_percent
)
select A.* from
(select order_no ,
item_id ,
sales_date ,
customer_id ,
city_id ,
qty ,
price ,
cogs ,
discount_percent from dbo.sales_stg ) A
left Join dbo.sales f on a.order_no = f.order_no
where f.order_no is null ;
update dbo.incr_info
set collectdate = (select max(sales_date ) from dbo.sales_stg)
where tablename1 = 'sales';
end
GIT File link - https://github.com/amitchandakpbi/powerbi/raw/main/Sales%20Data%20for%20Fabric.xlsx
All files: https://github.com/amitchandakpbi/
Power BI - Azure Free Tier, Create Azure Data Lake Gen 2 and connect that in Power BI: https://www.youtube.com/watch?v=x_hxpTgMAC0
Get Azure SQL DB and Configure it on Power BI: https://youtu.be/ZPfszCv3K3Y
Official Document: https://learn.microsoft.com/en-us/fabric/get-started/fabric-trial
Video from GuyinaCube: https://www.youtube.com/watch?v=5jlP0wdEsls
Buy a Microsoft Fabric subscription: https://learn.microsoft.com/en-us/fabric/enterprise/buy-subscription
SKU: https://learn.microsoft.com/en-us/fabric/enterprise/licenses#capacity-and-skus
Licenses: https://learn.microsoft.com/en-us/fabric/enterprise/licenses
Enable Fabric: https://learn.microsoft.com/en-us/fabric/admin/fabric-switch
🔵 What is Microsoft Fabric
Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence. It offers a comprehensive suite of services, including data lake, data engineering, and data integration, all in one place.
With Fabric, you don't need to piece together different services from multiple vendors. Instead, you can enjoy a highly integrated, end-to-end, and easy-to-use product that is designed to simplify your analytics needs.
How to enable Fabric, find details - https://learn.microsoft.com/en-us/fabric/admin/fabric-switch
Microsoft Fabric licenses details -https://learn.microsoft.com/en-us/fabric/enterprise/licenses
Don't forget to like, comment, and subscribe for more great content!
▶️Please Subscribe: https://www.youtube.com/@AmitChandak
▶️Data:
• Covid 19: https://covid19.who.int/WHO-COVID-19-...
• Get The data and Pbix files on Github: https://github.com/amitchandakpbi/powerbi
• DropBox files: https://www.dropbox.com/sh/90ycq7hpmngabe9/AABR_l_4sy3SSALA_U95gIkWa
---------
▶️Follow me on:
• Medium: https://amitchandak.medium.com
• Twitter: https://twitter.com/amitchandak78
• LinkedIn: https://www.linkedin.com/in/amitchandak78/
---------
▶️My Other Videos:
• Mastering Power BI: https://www.youtube.com/watch?v=wvsAzTqSDVg&list=PLPaNVDMhUXGaaqV92SBD5X2hk3TMNlHhb
• Expertise Power BI: https://www.youtube.com/watch?v=59PUFuuCrbY&list=PLPaNVDMhUXGYo50Ajmr4SgSV9HIQLxc8L
-~-~~-~~~-~~-~-
Please watch: "Microsoft Power BI Tutorial For Beginners✨ | Power BI Full Course 2023 | Learn Power BI"
https://www.youtube.com/watch?v=cN8AO3_vmlY
-~-~~-~~~-~~-~-
Видео Microsoft Fabric: Incremental ETL for Warehouse using Dataflow Gen 2, SQL Procedure, Data Pipeline канала Amit Chandak Learn Microsoft Fabric, Power BI, SQL
00:00 Overview of Incremental ETL
10:00 Create Warehouse and Load Data
26:00 What is the issue how to fix it
33:00 Load Data Using SQL procedure
36:30 Create Pipeline
30:00 Load Incremental Data
CREATE TABLE [dbo].[incr_info]
(
[tablename1] [varchar](4000) NULL,
[collectdate] [datetime2](6) NULL
)
insert into [dbo].[incr_info] values('sales', '2018-01-01');
CREATE TABLE dbo.sales_stg
(
order_no bigint,
item_id bigint,
sales_date datetime2(6),
customer_id bigint,
city_id bigint,
qty float,
price float,
cogs float,
discount_percent float
)
;
CREATE TABLE dbo.sales
(
order_no bigint,
item_id bigint,
sales_date datetime2(6),
customer_id bigint,
city_id bigint,
qty float,
price float,
cogs float,
discount_percent float
)
;
CREATE PROCEDURE dbo.loadSales
AS
BEGIN
insert into dbo.sales(
order_no ,
item_id ,
sales_date ,
customer_id ,
city_id ,
qty ,
price ,
cogs ,
discount_percent
)
select A.* from
(select order_no ,
item_id ,
sales_date ,
customer_id ,
city_id ,
qty ,
price ,
cogs ,
discount_percent from dbo.sales_stg ) A
left Join dbo.sales f on a.order_no = f.order_no
where f.order_no is null ;
update dbo.incr_info
set collectdate = (select max(sales_date ) from dbo.sales_stg)
where tablename1 = 'sales';
end
GIT File link - https://github.com/amitchandakpbi/powerbi/raw/main/Sales%20Data%20for%20Fabric.xlsx
All files: https://github.com/amitchandakpbi/
Power BI - Azure Free Tier, Create Azure Data Lake Gen 2 and connect that in Power BI: https://www.youtube.com/watch?v=x_hxpTgMAC0
Get Azure SQL DB and Configure it on Power BI: https://youtu.be/ZPfszCv3K3Y
Official Document: https://learn.microsoft.com/en-us/fabric/get-started/fabric-trial
Video from GuyinaCube: https://www.youtube.com/watch?v=5jlP0wdEsls
Buy a Microsoft Fabric subscription: https://learn.microsoft.com/en-us/fabric/enterprise/buy-subscription
SKU: https://learn.microsoft.com/en-us/fabric/enterprise/licenses#capacity-and-skus
Licenses: https://learn.microsoft.com/en-us/fabric/enterprise/licenses
Enable Fabric: https://learn.microsoft.com/en-us/fabric/admin/fabric-switch
🔵 What is Microsoft Fabric
Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence. It offers a comprehensive suite of services, including data lake, data engineering, and data integration, all in one place.
With Fabric, you don't need to piece together different services from multiple vendors. Instead, you can enjoy a highly integrated, end-to-end, and easy-to-use product that is designed to simplify your analytics needs.
How to enable Fabric, find details - https://learn.microsoft.com/en-us/fabric/admin/fabric-switch
Microsoft Fabric licenses details -https://learn.microsoft.com/en-us/fabric/enterprise/licenses
Don't forget to like, comment, and subscribe for more great content!
▶️Please Subscribe: https://www.youtube.com/@AmitChandak
▶️Data:
• Covid 19: https://covid19.who.int/WHO-COVID-19-...
• Get The data and Pbix files on Github: https://github.com/amitchandakpbi/powerbi
• DropBox files: https://www.dropbox.com/sh/90ycq7hpmngabe9/AABR_l_4sy3SSALA_U95gIkWa
---------
▶️Follow me on:
• Medium: https://amitchandak.medium.com
• Twitter: https://twitter.com/amitchandak78
• LinkedIn: https://www.linkedin.com/in/amitchandak78/
---------
▶️My Other Videos:
• Mastering Power BI: https://www.youtube.com/watch?v=wvsAzTqSDVg&list=PLPaNVDMhUXGaaqV92SBD5X2hk3TMNlHhb
• Expertise Power BI: https://www.youtube.com/watch?v=59PUFuuCrbY&list=PLPaNVDMhUXGYo50Ajmr4SgSV9HIQLxc8L
-~-~~-~~~-~~-~-
Please watch: "Microsoft Power BI Tutorial For Beginners✨ | Power BI Full Course 2023 | Learn Power BI"
https://www.youtube.com/watch?v=cN8AO3_vmlY
-~-~~-~~~-~~-~-
Видео Microsoft Fabric: Incremental ETL for Warehouse using Dataflow Gen 2, SQL Procedure, Data Pipeline канала Amit Chandak Learn Microsoft Fabric, Power BI, SQL
Показать
Комментарии отсутствуют
Информация о видео
17 июня 2023 г. 19:13:48
00:42:43
Другие видео канала
![#shorts Ultimate Free Power BI resources/Videos #learnpowerbi](https://i.ytimg.com/vi/p5qr2Y2o2Hw/default.jpg)
![Microsoft Fabric: Do k-Means clustering in spark; analyze in Power BI | Integrated Machine Learning](https://i.ytimg.com/vi/lsnvQ4UoTwo/default.jpg)
![Power BI Beginners 205: Power Query Table.CombineColumnsToRecord](https://i.ytimg.com/vi/NaA5PHwgsKE/default.jpg)
![Power BI (हिंदी) March 2023- Visual on-Object, Apply all, Clear All](https://i.ytimg.com/vi/poxwf-zJHwA/default.jpg)
![Power BI Tutorial: Sync Slicers Across Different Tables and Pages| Sync different columns](https://i.ytimg.com/vi/7pgoaZWfMOk/default.jpg)
![Power BI: Dynamic TOPN: Window, Field parameters, calculation groups](https://i.ytimg.com/vi/YW1SOk9V9xk/default.jpg)
![Power BI Abstract Thesis Episode 45: Tenant Settings- Audit and Usage Settings](https://i.ytimg.com/vi/3C7iySjT4-Q/default.jpg)
![#powerbitutorial #powerbininterview Your Power BI Interview 2024! Top 100 Must-Know Questions](https://i.ytimg.com/vi/xQMuFcCqZ_M/default.jpg)
![Power BI Tutorial Series for Beginners 131: Power Query- Add Prefix and add suffix](https://i.ytimg.com/vi/RgazgGYDqPQ/default.jpg)
![Power BI - New DAX Function: Rownumber](https://i.ytimg.com/vi/yS9-IQjUDwg/default.jpg)
![Power BI Tutorial Series for Beginners 168: Power Query- List.IsEmpty](https://i.ytimg.com/vi/ouHLrWFJyds/default.jpg)
![#shorts #powerbi know your data better- Column Profile, Distribution and Quality| Power Query](https://i.ytimg.com/vi/iTQdsi9ZagE/default.jpg)
![Learn Tableau With Tableau Public 21: LOD Include](https://i.ytimg.com/vi/Rcs0REbg59o/default.jpg)
![Learn Power BI: Window as Rank- why and Why not](https://i.ytimg.com/vi/Pu8y8bd5mb8/default.jpg)
![Learn Power BI 217: Table HasColumns, PrefixColumns, DuplicateColumn](https://i.ytimg.com/vi/A_-Rh0605Nc/default.jpg)
![Microsoft Fabric: Load data in Warehouse- Dataflow Gen2| Note- Warehouse Destination available now](https://i.ytimg.com/vi/Mn1GB8rSeoQ/default.jpg)
![Abstract Thesis 97: Power BI- Power Query: When I asked you to create common tables](https://i.ytimg.com/vi/PqfGW6pl1Sw/default.jpg)
![Microsoft Fabric in Hindi: How to load data in Lakehouse using Dataflow Gen 2 | End to End Scenario](https://i.ytimg.com/vi/vJi5kQdSe2I/default.jpg)
![Microsoft Fabric: Shortcut within Fabric and for Azure Data Lake| File vs Table Shortcut](https://i.ytimg.com/vi/KIbt9GzAdWk/default.jpg)
![SQL Tutorial 10: Union, Union All, Intersect, and Except/Minus](https://i.ytimg.com/vi/NQPe_tPhf5I/default.jpg)