Quality Assurance in Beam: measure your pipeline! (Beam Summit Europe 2019)

Apache Beam being a Unified Model supports multiple Runners and SDKs. You also want it to work like a charm on multiple file systems and be sure that any IO is fine with big loads. Other than that you need your jobs to be efficient (and you probably want to know “all the numbers”). How to ensure that all your goals are met? You need proper testing for that! However, due to the complex nature of the problem itself, automated testing can sometimes be tricky too!

Luckily, we’ve provided a whole bunch of tools to Beam's codebase to ease your pain. In this presentation, we will show you how you can write various integration/performance tests in Beam using an auto-setup infrastructure (DBs, filesystems & runners), BigQuery and Metrics API. We will also show you how do we currently do it in Beam in both IO Integration Tests or Load Tests of core Beam transforms.

Use these tools to increase the quality of your beam pipelines!

Speakers:
Łukasz Gajowy - Software Engineer at Polidea, Beam Committer
Michał Walenia - Software Developer at Polidea
Katarzyna Kucharczyk - Software Engineer at Polidea

The Beam Summit Europe 2019 was a 2 day event held in Berlin at the KulturBrauerei, all focused around Apache Beam.

For more information about the Beam Summit, follow us on twitter @BeamSummit or go to the website: https://beamsummit.org/

Видео Quality Assurance in Beam: measure your pipeline! (Beam Summit Europe 2019) канала Apache Beam