3 ms·Web-scale crawling at cable modem speeds8 points by jmischel 16y agoA high-level overview of why and how we use machine learning techniques to focus our crawling, giving us wide coverage of online video with limited bandwidth.