Apache Spark : Commonly used Transformations : Map, Filter, Flatmap Transformations

Official Website: http://bigdataelearning.com

Learning Objectives :: In this module, you will learn some of the commonly used transformations. You will learn some of the basic RDD transformations like Map, Filter, and Flatmap transformations.

This video also shows how to apply the Map, Filter, and Flatmap transformation using Scala & python.

Topics :: Commonly used Transformations, Basic RDD Transformations, Map, Filter, Flatmap Transformations

Commonly used Transformations
=============================

Basic RDD Transformations,
=========================
map, filter, and flatmap are some of the basic RDD transformations

Map
====
Map is a transformation , that takes a function and applies the function to each elements of the input RDD.

The result of the function , will become the value of each element , in the resultant RDD.

say, If inputRDD contains values 1 to 4, then map transformation to square the values will return {1,4,9,16} as the resultant RDD
Here the square function is applied to each elements of the inputRDD

Filter
====
Filter is a transformation , that returns a new RDD , with only the elements that passes the filter condition.

say, if inputRDD contains values 1 to 3, then applying transformation to filter elements that are not '1' , will return only '2' and '3' as the resultant RDD.

Flatmap
=======
Flatmap() is the transformation that takes a function , and applies the function to each elements of the RDD as in map() function.
The difference is that flatmap will return multiple values for each element in the source RDD.

say, if inputRDD contains the values {"hello world" and "how are you"} then applying split function, to flatmap transformation, will
return an array of words like {"hello","world","how","are","you"}. Since the flatmap transformation returns multiple values for each element, there are 5 elements in the resultantRDD, where as inputRDD has only 2 elements.

To recollect , if we apply the split function to Map transformation instead of Flatmap transformation, then we will get the values as the highlighted ones.

Here the words will be splitted into multiple words, however the words belonging to an element of source RDD is still a single element in the resultant RDD.

Видео Apache Spark : Commonly used Transformations : Map, Filter, Flatmap Transformations канала BigDataElearning