AWShorts EP03 - Serverless Data Pipeline (NYC Taxi Dataset → S3 → Lambda→Glue → Athena → QuickSight)

In this episode I create a data lake / serverless pipeline using a NYC dataset, S3, Lambda, Glue, Athena & QuickSight

Using S3 to store the dataset, once uploaded, Lambda will fully trigger a pipeline, the data will be sent to Glue first to automatically detect the schema and prepare the data. I'll use Athena where I can do queries in the data and finally it will go to QuickSight to visualise the data

Dataset used: https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page
GitHub Repo: https://github.com/JohnMichaelCrawley/AWShorts/tree/main/EP03-Serverless-Data-Pipeline%28NYC-dataset-S3-Glue-Athena-Quicksight%29

Side note:
I was meant to upload this sooner, but neighbours decided to build an extension making it much harder to record the voice over for this episode

Intro music:
Artist: Kevin MacLeod
Song: Night in Venice

Видео AWShorts EP03 - Serverless Data Pipeline (NYC Taxi Dataset → S3 → Lambda→Glue → Athena → QuickSight) канала John Crawley