Infrastructure for orchestrating dlt pipelines workshop by The Scalable Way

Pipelines working...but platform missing?

A hands-on workshop to turn your early-stage data workflows into a structured, scalable platform.

Learn to productize your data platform and orchestrate dlt pipelines by The Scalable Way. This hands-on workshop covers lightweight infrastructure, CI/CD, and flow automation, giving you practical steps to build a scalable and maintainable environment for real-world data workflows.

Start here to see who The Scalable Way are, what they do, and how they can help: https://thescalableway.com/

Follow The Scalable Way on LinkedIn for fresh insights, event updates, and real-world platform wins: https://www.linkedin.com/company/scalable-way

Dive deeper into Karol’s talk with their productized data platform blog series: https://thescalableway.com/tags/productized-data-platform/

Test your scalability in minutes and get tailored recommendations: https://thescalableway.scoreapp.com/

Stay updated—subscribe for new articles, templates, and invites: https://thescalableway.com/subscribe/

Connect with Karol on LinkedIn—questions, feedback, and ideas welcome: https://www.linkedin.com/in/wolski-karol/

View the slides to revisit frameworks, patterns, and links from the session: https://docs.google.com/presentation/d/1r_kT70_3ENdR1aPjpa5yZpiAqQsVAY-NPdbi9hu3x08/edit?usp=sharing

LLM Generated Description:

Unlock the full potential of your data infrastructure with our in-depth workshop on orchestrating dlt pipelines. Delve into the practicalities of building robust, scalable data platforms using dlt alongside renowned orchestration tools like Prefect and Airflow. Perfect for experienced data engineers ready to streamline their workflows, this session covers essential steps for a successful deployment, from initial pipeline setup to advanced scaling techniques.

* Learn how to set up a modern data platform leveraging dlt for seamless data ingestion and transformation.
* Explore the benefits of using Prefect over Airflow for ELT workflows, including its Pythonic approach and managed/hosted options.
* Gain insights into infrastructure as code using Terraform for consistent and scalable deployment across cloud environments.
* Discover practical orchestration with Kubernetes and Prefect to manage, scale, and monitor your data workflows effectively.
* Dive into the challenges of server-based vs. serverless architectures and the strategies to maintain cost efficiency and control.
* Understand the importance of CI/CD pipelines specifically tailored for data platforms, emphasizing automation and reproducibility.

Watch now to revolutionize your data pipeline orchestration and maximize the capabilities of your data platform. Transform API data into actionable insights with cutting-edge tools like dbt — https://dlthub.com/docs/dlt-ecosystem/transformations/dbt.

*Related links*
Build a pipeline (tutorial) — https://dlthub.com/docs/build-a-pipeline-tutorial/
Transform with SQL — https://dlthub.com/docs/dlt-ecosystem/transformations/sql
Airflow / Cloud Composer — https://dlthub.com/docs/walkthroughs/deploy-a-pipeline/deploy-with-airflow-composer
Prefect — https://dlthub.com/docs/walkthroughs/deploy-a-pipeline/deploy-with-prefect
GitHub Actions — https://dlthub.com/docs/walkthroughs/deploy-a-pipeline/deploy-with-github-actions

Start using dlt today!
Docs: https://dlthub.com/docs/intro/
Learn: https://dlthub.learnworlds.com/

Видео Infrastructure for orchestrating dlt pipelines workshop by The Scalable Way канала dltHub