Beam on Flink: How does it actually work? - Maximilian Michels
Apache Beam is a data processing model built with focus on portability. Beam jobs can be written in the language of your choice: Java, Python, Go, or SQL. Once written, they can be executed using various execution engines including Apache Flink, Apache Spark, Google Cloud Dataflow, and many more.
In order for Beam to support multiple execution engines, the Beam API needs to be translated to the API of the execution engine (e.g. Flink's). In Beam, this is the responsibility of the ""Runner"".
The Flink Runner has come a long way from an early stage Runner to a fully-featured Runner. Its latest addition is the integration of Beam's language portability layer which enabled to run jobs written in other languages than Java.
In this talk, we will dissect the Flink Runner and show how Beam's components tie together with Flink. If you have ever wondered how the Flink Runner or Beam works, this is your chance to find out.
Видео Beam on Flink: How does it actually work? - Maximilian Michels канала Flink Forward
In order for Beam to support multiple execution engines, the Beam API needs to be translated to the API of the execution engine (e.g. Flink's). In Beam, this is the responsibility of the ""Runner"".
The Flink Runner has come a long way from an early stage Runner to a fully-featured Runner. Its latest addition is the integration of Beam's language portability layer which enabled to run jobs written in other languages than Java.
In this talk, we will dissect the Flink Runner and show how Beam's components tie together with Flink. If you have ever wondered how the Flink Runner or Beam works, this is your chance to find out.
Видео Beam on Flink: How does it actually work? - Maximilian Michels канала Flink Forward
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
How to build a modern stream processor: The science behind Apache Flink - Stefan RichterIntroduction to Stateful Stream Processing with Apache Flink • Robert Metzger • GOTO 2019Streaming Concepts & Introduction to Flink - Use Case: Event-Driven ApplicationsHow Lyft built a streaming data platform with Flink on Kubernetes - Micah WyldeCan we make quantum technology work? | Leo Kouwenhoven | TEDxAmsterdamApache Beam Explained in 12 MinutesWebinar: Deploying Flink on Kubernetes - David AndersonSonny Scroggin - BEAM + Rust: A match made in heaven | Code BEAM STO 19Apache Flink Worst Practices - Konstantin KnaufDynamically Generated Flink Jobs at Scale - Regina Chan, Goldman SachsThe Evolution Of Stealth TechnologyStreaming Analytics Made Easy: Kinesis Data Analytics Studio Run on Apache FlinkStreaming Concepts & Introduction to Flink Series - What is Stream Processing & Apache Flink"Building a Python Data Pipeline with Apache Flink" - Caito Scherr (PyCascades 2021)gRPC IntroductionFacebook and memcached - Tech TalkEfficient Window Aggregation with Stream Slicing - Jonas Traub & Philipp GrulichBig Data Processing with Apache Beam Python | SciPy 2017 | Robert BradshawRobust Stream Processing with Apache FlinkTrading at market speed with the latest Kafka features by Iñigo González