4 ms·
> scrapers slow down their site and that's why they use robots.txt a poorly written scraper may really slow down your site, especially if it wasn't intended to
by chii 10y ago
> scrapers slow down their site and that's why they use robots.txt
a poorly written scraper may really slow down your site, especially if it wasn't intended to be scrapped repeatedly. There should be something to be said about frequency which scrapers should follow (specified by the website owner via a robots.txt like spec).
But website owners cannot demand unreasonable frequencies (such as once a year!), and what constitutes unreasonable is up for debate.
- davb 10y agoAdditionally, if excessive scraping became an issue for my site I'd consider rate limiting client.
- timClicks 10y agoThe Crawl-delay directive is the de facto standard for this.
- eknkc 10y agoI don't think a poorly written scraper would follow robots.txt rules according to spec. So, in any case the site should have other measures (rate limiting?) anyway.
- dingo_bat 10y ago> (specified by the website owner via a robots.txt like spec). Nope, if a website wants such a restriction, it must enforce it. Robots.txt is a request. It's worthless.
- bkor 10y agoIf a robot misbehaves, it'll either be blocked or it'll go to the networks abuse section and that bot will be taken down. That a site possibly could have some kind of technical solution to this doesn't matter.
- problems 10y agoPrecisely - the solution here needs to be that the server blocks the robot - if it can differentiate it from other traffic that is. That's all well and good and that's the solution which should be used here. If you don't want to be archived, block the IP.