Running a Massively Parallel Self serve Distributed Data System At Scale

Keystone is the critical piece of Netflix backend infrastructure to ensure massive amount of events are processed in near real time, reliably, at scale, and in face of failures in a cloud-native microservices environment. Turns out, such an embarrassingly parallel stream processing system is not embarrassingly easy to develop and operate, especially given the challenges of unpredictable failures in a cloud-native environment, self-serve multi-tenancy support, and assumptions of maintaining extremely high development/operation agility.

In this talk, Zhenzhong Xu will shed light on how we built an elastic, resilient, reactive, and self-healing distributed system in the cloud.

* High-level cloud-native microservice based Keystone architecture.
* A deep dive on how we built the system based on ideas such as declarative reconciliation, container based immutable deployment, logical workload isolation, and chaos exercise.
* Insights into our operation best practices, such as capacity provisioning, delivery semantics, deployment tradeoffs, back-pressure management, etc.

Видео Running a Massively Parallel Self serve Distributed Data System At Scale канала Netflix Data