Building a scalable focused web crawler with Flink - Ken Krugler

Flink Forward San Francisco, April 2018 #flinkforward

Building a scalable focused web crawler with Flink - Ken Krugler

Is it possible to build an efficient, focused web crawler using Flink? That was the question that led to the creation of the flink-crawler open source project. In this talk I’ll discuss how we use Flink’s support for AsyncFunctions and iterations to create a scalable web crawler that continuously and efficiently performs a focused web crawl with no additional infrastructure. I’ll also discuss some of the testing and debugging challenges encountered when using features such as AsyncFunctions and iterations.

https://data-artisans.com/

Видео Building a scalable focused web crawler with Flink - Ken Krugler канала Flink Forward