Atrax, a distributed web crawler
This talk describes Atrax, a distributed and very fast web crawler. Running Atrax on a cluster of four DS20E Alpha servers saturates our internet connection. During a recent crawl, we were able to download about 115 Mbits/sec, or about 50 million web pages per day, over a sustained period of time. Atrax has been used to collect the raw data for numerous web studies performed at Compaq Research.
Видео Atrax, a distributed web crawler канала Microsoft Research
Видео Atrax, a distributed web crawler канала Microsoft Research
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
![The Intern Experience at Microsoft Research Cambridge](https://i.ytimg.com/vi/qiCcJBZm8FY/default.jpg)
![Battling Tuberculosis Using Microsoft Technology](https://i.ytimg.com/vi/r2w40ZKxVUg/default.jpg)
![Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes](https://i.ytimg.com/vi/apayUKSExmU/default.jpg)
![Data Visualization Reaches New Heights with Layerscape](https://i.ytimg.com/vi/Y4C2tvteQU4/default.jpg)
![Faculty Summit 2018 Introduction](https://i.ytimg.com/vi/RnzjxXOqovc/default.jpg)
![Microsoft Pix - Take Better Photos of People, Automatically](https://i.ytimg.com/vi/QXH_UaWMV48/default.jpg)
![Sirius: A Flat Datacenter Network with Nanosecond Optical Switching (SIGCOMM 2020 short talk)](https://i.ytimg.com/vi/y5Op0bhoLT8/default.jpg)
![Improvements on Higher Order Ambisonics Reproduction](https://i.ytimg.com/vi/DAlAaVTO8kg/default.jpg)
![Cambridge lab overview with Chris Bishop](https://i.ytimg.com/vi/8dR5lPav6K0/default.jpg)
![Get free cloud computing time and storage on Microsoft Azure](https://i.ytimg.com/vi/OLcOugO4Rqk/default.jpg)
![How interns at our New England Lab impact research at Microsoft](https://i.ytimg.com/vi/RNwIkn752uE/default.jpg)
![AI Institute "Geometry of Deep Learning" 2019 [Workshop] Day 1 | Session 2](https://i.ytimg.com/vi/FCmUtpkDk-I/default.jpg)
![De-Identifying Healthcare Data for Research](https://i.ytimg.com/vi/h-VhEVlC3h0/default.jpg)
![Intelligent cloud computing lifts villages out of water poverty](https://i.ytimg.com/vi/FCWMe1snMP8/default.jpg)
![How Microsoft and Novartis created Assess MS (short version)](https://i.ytimg.com/vi/R0dRuQ7Xy2w/default.jpg)
![Project Prague - The Producer - Behind the Scenes](https://i.ytimg.com/vi/UIakBZfEpPA/default.jpg)
![How OSIsoft and Deschutes Brewery used Microsoft Security Risk Detection](https://i.ytimg.com/vi/zMm3sUOm9jw/default.jpg)
![ChronoZoom curriculum and technology](https://i.ytimg.com/vi/awWCR81Wtdc/default.jpg)
![Recent Advances in Image Captioning, Image-Text Retrieval and…](https://i.ytimg.com/vi/4wS02nCWXvw/default.jpg)
![Using machine learning and AI to reduce hospital readmissions](https://i.ytimg.com/vi/bV8FHKCTx5k/default.jpg)
![IROS 2020 - Mixed Reality and Robotics Tutorial - Demo 1: Interaction](https://i.ytimg.com/vi/4G3wjPIs4Fc/default.jpg)