Achieving Operational Excellence for data engineering - Shantanu Shirvastarva, Zeotap

Zeotap saw a 10x growth with the number of data pipelines and amount of data processed growing at rapid pace in a short span of time. The increasing scale challenged our capability to track production data pipelines across multiple products and infrastructure deployments. To tackle this problem, we came up with Kingpin - a Centralized Job Management Framework. Kingpin addresses some of the common issues faced with data pipeline operations: Workflow Dependency management, Multi-Framework execution support Fault Tolerance, and Alerting mechanisms, Scheduled and On-Demand executions Workflow tracking across multiple deployments. Additionally, we have integrated Kingpin with Data Visualization Tools for a Unified Job Level View with Metric Reporting capabilities to increase system visibility and reduce the operational time, efforts, and costs.

Видео Achieving Operational Excellence for data engineering - Shantanu Shirvastarva, Zeotap канала Qubole: The Cost-Efficient Data Lake