Data Pipelines with Python and PostgreSQL
Code : https://github.com/Sean-Bradley/Stream-Data-From-Flask-To-Postgres
I show how to use streaming techniques to build a data pipeline which pulls data from an external API that returns massive amounts of data, and insert it straight into a PostgreSQL data base ASAP.
The Python process that reads the huge amount of data, before inserting into Postgres is able to process the incoming chunks by use of the stream=True option on the Requests module, and the iter_content method of the Request response. The mock 3rd party API, which hosts the potentially petabytes of data, makes use of the Flask stream_with_context method.
Видео Data Pipelines with Python and PostgreSQL канала seanwasere ytbe
I show how to use streaming techniques to build a data pipeline which pulls data from an external API that returns massive amounts of data, and insert it straight into a PostgreSQL data base ASAP.
The Python process that reads the huge amount of data, before inserting into Postgres is able to process the incoming chunks by use of the stream=True option on the Requests module, and the iter_content method of the Request response. The mock 3rd party API, which hosts the potentially petabytes of data, makes use of the Flask stream_with_context method.
Видео Data Pipelines with Python and PostgreSQL канала seanwasere ytbe
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
![Data Engineering Principles - Build frameworks not pipelines - Gatis Seja](https://i.ytimg.com/vi/pzfgbSfzhXg/default.jpg)
![Building data pipelines with Kafka and PostgreSQL](https://i.ytimg.com/vi/Ca3TlklRhfs/default.jpg)
![ETL with Python](https://i.ytimg.com/vi/InLgSUw_ZOE/default.jpg)
![](https://i.ytimg.com/vi/39GntJcG894/default.jpg)
![Database Indexing Explained (with PostgreSQL)](https://i.ytimg.com/vi/-qNSXK7s7_w/default.jpg)
![What is Data Pipeline | How to design Data Pipeline ? - ETL vs Data pipeline](https://i.ytimg.com/vi/VtzvF17ysbc/default.jpg)
![Python tricks: Demystifying async, await, and asyncio](https://i.ytimg.com/vi/tSLDcRkgTsY/default.jpg)
![Step-by-Step Python and Postgres Tutorial with psycopg2](https://i.ytimg.com/vi/2PDkXviEMD0/default.jpg)
![Data Pipeline Frameworks: The Dream and the Reality | Beeswax](https://i.ytimg.com/vi/C6Abv87D5dU/default.jpg)
![5 Data Engineering Projects Ideas To Put On Your Resume](https://i.ytimg.com/vi/385mKftVr3I/default.jpg)
![How to Design and Build a Recommendation System Pipeline in Python (Jill Cates)](https://i.ytimg.com/vi/v_mONWiFv0k/default.jpg)
![WHAT IS DATA PIPELINE? | BEST TOOLS FOR OPERATIONS WITH DATA PIPELINES](https://i.ytimg.com/vi/0qJgz5A3ND8/default.jpg)
![Docker Volumes with PostgreSQL Example](https://i.ytimg.com/vi/G-5c25DYnfI/default.jpg)
![Pandas Python Tutorial: Creating a Pipeline in Pandas](https://i.ytimg.com/vi/v1kMoh2IAmY/default.jpg)
![AWS Data Pipeline Tutorial | AWS Tutorial For Beginners | AWS Certification Training | Edureka](https://i.ytimg.com/vi/5eq6fiw1dPA/default.jpg)
![Python MongoDB Tutorial using PyMongo](https://i.ytimg.com/vi/rE_bJl2GAY8/default.jpg)
![Creating Pipelines Using SKlearn- Machine Learning Tutorial](https://i.ytimg.com/vi/w9IGkBfOoic/default.jpg)
![A simple Dataflow pipeline (Python)](https://i.ytimg.com/vi/J-b2Eo5Qvp8/default.jpg)
![Quick Python 6: Our First Data Pipeline](https://i.ytimg.com/vi/VCqwKazLLbY/default.jpg)
![SQLAlchemy: Object Relational Modelling (ORM) for Python, quick-start](https://i.ytimg.com/vi/OT5qJBINiJY/default.jpg)