Data Engineering in the Age of LLMs: The New Frontier

Large Language Models (LLMs) are revolutionizing the tech world, and Data Engineering is no exception. In this tutorial, we explore the paradigm shift in data engineering tasks brought about by LLMs. We'll dive into new concepts like embedding pipelines, vector databases, document chunking, and prompt caching. We will also build a simple RAG (Retrieval-Augmented Generation) pipeline using Python, LangChain, and a mock data warehouse to demonstrate how to unlock insights from your private data using LLMs. Join us to learn the essential skills for the modern data engineer!

#dataengineering #llm #ai #langchain #vectordatabase #python #bigdata #datapipelines

Видео Data Engineering in the Age of LLMs: The New Frontier канала Data Engineering with Subhadip