Data Engineering Principles - Build frameworks not pipelines - Gatis Seja
PyData London Meetup #54
Tuesday, March 5, 2019
Data pipelines are necessary for the flow of information from its source to its consumers, typically data scientists, analysts and software developers. Managing data flow from many sources is a complex task where the maintenance cost limits scale of being able to build a large reliable data warehouse. This presentation proposes a number of applied data engineering principles that can be used to build robust easily manageable data pipelines and data products. Examples will be shown using Python on AWS.
Sponsored & Hosted by Man AHL
****
www.pydata.org
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.
PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases.
Видео Data Engineering Principles - Build frameworks not pipelines - Gatis Seja канала PyData
Tuesday, March 5, 2019
Data pipelines are necessary for the flow of information from its source to its consumers, typically data scientists, analysts and software developers. Managing data flow from many sources is a complex task where the maintenance cost limits scale of being able to build a large reliable data warehouse. This presentation proposes a number of applied data engineering principles that can be used to build robust easily manageable data pipelines and data products. Examples will be shown using Python on AWS.
Sponsored & Hosted by Man AHL
****
www.pydata.org
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.
PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases.
Видео Data Engineering Principles - Build frameworks not pipelines - Gatis Seja канала PyData
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
Future of Data EngineeringWhat is a Data Engineer?Data Pipeline Frameworks: The Dream and the Reality | BeeswaxRob Story | Data Engineering Architecture at SimpleFunctional Data Engineering - A Set of Best Practices | LyftDelivering High Quality Analytics at NetflixReal-Time Data Pipelines Made Easy with Structured Streaming in Apache Spark | DatabricksPrefect as a Data Engineering Project Workflow Tool, with Mary Clair Thompson (Duke) - 11/6/2020Airflow in Practice Stop Worrying Start Loving DAGs - Sarah SchattschneiderAmazon Data Engineer-1 Interview Experience | In Covid Times | Bar Raiser | Questions in Each Round🔥Data Engineering and Data Science: Bridging the Gap | DataEDGE 2016James Powell: So you want to be a Python expert? | PyData Seattle 2017What is the interview process for a Data Engineering position - My experienceFour Distributed Systems Architectural Patterns by Tim BerglundBuilding data pipelines with Kafka and PostgreSQLWhat is Data Pipeline | How to design Data Pipeline ? - ETL vs Data pipelineWhat do DATA ENGINEERS do? Is data engineering a good career choice in 2020?James Powell: Design Principles | PyData DC 2016