3 ms·
People who come to Twitter with an intent to scrape it aren't going to look at robots.txt.
by dheera 3y ago
People who come to Twitter with an intent to scrape it aren't going to look at robots.txt.
- extraduder_ire 3y agoNot sure it applies to twitter specifically, but if you are scraping websites you should check robots.txt. The intended purpose is to direct you to where the content actually is, so you're not wasting resources spidering through contentless pages, just because they're linked from somewhere.
- falcolas 3y agoFunny enough, the some of the biggest scrapers in the world do respect robots.txt - search engine scrapers, to be specific.
- dheera 3y agoThere's a big difference between A. "I'm going to scrape the web to do X, Y, Z" and B. "I'm going to scrape Twitter data to do X, Y, Z" A is far more likely to care about robots.txt because there's enough left of the web to complete their mission. B will not care if Twitter has a robots.txt file.