4 ms·
Funny enough, the some of the biggest scrapers in the world do respect robots.txt - search engine scrapers, to be specific.
by falcolas 3y ago
Funny enough, the some of the biggest scrapers in the world do respect robots.txt - search engine scrapers, to be specific.
- dheera 3y agoThere's a big difference between A. "I'm going to scrape the web to do X, Y, Z" and B. "I'm going to scrape Twitter data to do X, Y, Z" A is far more likely to care about robots.txt because there's enough left of the web to complete their mission. B will not care if Twitter has a robots.txt file.