3 ms·
A little web scraping is fine, if and only if the website lacks a canonical way to do so (i.e. an API). Ignoring the API and scraping hard enough that you have
by minimaxir 10y ago
A little web scraping is fine, if and only if the website lacks a canonical way to do so (i.e. an API). Ignoring the API and scraping hard enough that you have to write an article assessing scalability of the scraping is bad.
- cglee 10y agoIt's weird that you're dictating rules around scraping not outlined in Github's TOS. The article actually mostly talks about how they scaled working with the accumulated data, and only a little about scaling the scraper. Just like the interesting part about Google is how they process and index the data and not their crawler, the most interesting part of this article actually not the scraping, but how they handled all the data and processed it. Edit: I found Github's policy on scraping. I hope the link brings some closure to this concern: https://help.github.com/articles/github-terms-of-service-draft/#5-scraping https://help.github.com/articles/github-terms-of-service-dra...
- michaelrm 10y agoA bit more context may be helpful. While scraping scalability was a concern for us (for latency reasons), it was only significant to retrieve historical data as a one-time job, after which we throttled back. A latter segment talks about a priority queue object (SRRPQQ) which we use in lieu of scaling further.