Netflix Data Engineering Tech Talks - Building Reliable Data Pipelines

Holden Karau, OSS Engineer, Data Platform Engineering, talks about the importance of reliable data pipelines and how to build them covering tools from testing to validation and auditing. The talk uses Apache Spark as an example, but the concepts generalize regardless of your specific tools.

Some related projects include:

https://github.com/holdenk/spark-testing-base
https://github.com/unionai-oss/pandera
https://github.com/target/data-validator
and
https://github.com/tensorflow/data-validation.

#netflix
#datascience
#dataengineering
#etl
#bigdata

Видео Netflix Data Engineering Tech Talks - Building Reliable Data Pipelines канала Netflix Data