Autotuning your Spark config in Microsoft Fabric (21 of 30)

Learn Apache Spark in Microsoft Fabric in the 30 days of September.

Here's the playlist for this series if you want to catchup: https://www.youtube.com/playlist?list=PLug2zSFKZmV1z1zUCs0iUEHgtmHJIwFiQ

Docs for Spark Autotune in Microsoft Fabric: https://learn.microsoft.com/en-us/fabric/data-engineering/autotune?tabs=sparksql

Link to the GitHub for this series: https://github.com/LearnMicrosoftFabric/YouTube/tree/main/learn_spark_in_fabric

Spark is the engine behind both the Data Engineering AND the Data Science experiences in Microsoft Fabric, so in September I'll be walking you through Apache Spark: what it is, why you should learn it, how to use it, and how it integrates into Microsoft Fabric.

No previous Spark knowledge is required, some basic Python would be useful!

#pyspark #microsoftfabric #apachespark

Here's the schedule:
1 Welcome - https://youtu.be/v5pJLJg9j6c
2 Why Spark? https://youtu.be/fFN7uvra8tg
3 Components of Spark - https://youtu.be/p3Ss_P6AyG0
4 Spark DataFrame - https://youtu.be/51EMJfnx47Q
5 Read Files into DataFrame - https://youtu.be/ja5MFgpdTAw
6 Read/Write to Lakehouse Table - https://youtu.be/bou77s-9b7c
7 Basic DataFrame Operations - https://youtu.be/tvM5MYgmips
8 DataFrame Filtering - https://youtu.be/rvgcJ81KYFY
9 GroupBy and Aggregate Functions - https://youtu.be/fuxlLQmuccY
10 Handling missing values - https://youtu.be/H83lW_RkeRU
11 Joining DataFrames - https://youtu.be/4bP5wqRH3AU
12 Time-series data - https://youtu.be/PlIUuuCtKm4
13 Spark SQL - https://youtu.be/eoj86nVraeI
14-16 Spark Machine Learning - https://youtu.be/f3-Xr3tPPsc
20 Configuring Spark - https://youtu.be/HIW8MvX4pWw
21 Autotuning Spark Configuration - https://youtu.be/PGbC_Z7Fhv0
22 Library Management - https://youtu.be/9pfBPXSRwlM
23 High-concurrency mode - https://youtu.be/4Fi1hzLNya4
24 Spark Scala - https://youtu.be/SuzmZszsZhc
25 Fabric MSSparkUtils - https://youtu.be/50z--o0R5dM
26 Monitoring Spark - https://youtu.be/CslPrMXxpUY
30 FINALE QnA - https://youtu.be/KmL5qhAGhAs

Timeline
0:00 Intro to Autotuning
1:10 Reviewing the docs
2:05 What does it optimise?
3:30 The Magic Oneliner
4:20 Benefits of autotuning
5:00 How does it work?
5:53 Wrapup

--BROWSE MY OTHER FABRIC PLAYLISTS--
DATA ENGINEERING https://www.youtube.com/playlist?list=PLug2zSFKZmV1NvKfnRzG9e3Fl-8QLD5MK
END-TO-END FABRIC PROJECT https://www.youtube.com/playlist?list=PLug2zSFKZmV1BHk129X_2xi53ZRwQaTkb
INTRO TO MICROSOFT FABRIC https://www.youtube.com/playlist?list=PLug2zSFKZmV0Yaya7NxRQfrrPtfF2vj0K
DATA FACTORY https://www.youtube.com/playlist?list=PLug2zSFKZmV3FkUFDxlyrfSJMQ3CeqyEF

--LINKEDIN--
Not following the LinkedIn page yet? Here's the link: https://www.linkedin.com/company/learnmicrosoftfabric/

--ABOUT WILL--
Hi, I'm Will! I'm hugely passionate about data and using it to create a better world. I currently work as a Consultant, focusing on Data Strategy, Data Engineering and Business Intelligence (within the Microsoft/Azure/Fabric environment). I have previously worked as a Data Scientist. I started Learn Microsoft Fabric to share my learnings on how Microsoft Fabric works and help you build your career and build meaningful things in Fabric.

--SUBSCRIBE--
Not subscribed yet? You should! There are lots of new videos in the pipeline covering all aspects of Microsoft Fabric.
https://youtube.com/@learnmicrosoftfabric?sub_confirmation=1

Видео Autotuning your Spark config in Microsoft Fabric (21 of 30) канала Learn Microsoft Fabric with Will