Build a Custom Transformer in Pyspark

In this video we will cover how to build a Custom Transformer in Pyspark. We will cover what Transformers are, why we should use them, and how they can be implemented, and incorporated, into Pyspark pipelines.

The break-down of the video is as follows:

Introduction 00:00
Use of Transformers 00:35
Jupyter Notebook Setup 01:33
Dataset & Feature Logic 02:08
Coding a Transformer in Pyspark 06:32
Testing in a Pipeline 15:56
Conclusions 21:13

The best way to keep up-to-date with my video/blog content is to sign up for my monthly Newsletter! Please visit: https://insidelearningmachines.com/newsletter/ to register.

The notebook presented here can be found at: https://github.com/insidelearningmachines/Blog/blob/main/Notebook%20XXVII%20Custom%20Pyspark%20Transformers.ipynb

This video is based off of an article on my blog. You can find that blog article here: https://insidelearningmachines.com/custom_transformer_in_pyspark/

The homepage of my blog is: https://insidelearningmachines.com

Other social media includes:
Twitter: https://twitter.com/inside_machines
Facebook: https://www.facebook.com/Inside-Learning-Machines-112215488183517

#machinelearning #datascience #apache-spark #pyspark #insidelearningmachines

Видео Build a Custom Transformer in Pyspark канала Inside Learning Machines