4 ms·
Yeah, I have nothing at all against scraping per se, it's more about the huge bot traffic the commercial scrapers generate, which would ALSO be fine, but 1. it
by sillyquiet 7y ago
Yeah, I have nothing at all against scraping per se, it's more about the huge bot traffic the commercial scrapers generate, which would ALSO be fine, but 1. it can be hard to tell scrapers from malicious DDOS bots sometimes and 2.) the person being scraped literally pays for that scraping traffic.
- a1369209993 7y agoSo, apply DOS/DDOS mitigations? You need to have those anyway, so you might as well use them for this too.
- sillyquiet 7y agoYeah, but where do you draw the line between 'oh sorry we dropped your request because of rate limiting' (or whatever mitigation strategy) and 'oh, we dropped your request because u scraping us bro' legally? IANAL, but this lawsuit seems to indicated that putting barriers in front of scraping attempts is a no-no.
- a1369209993 7y ago"You issued more than $X requests per $Y seconds. We don't care whether you're scraping or not; try again later." for any values of X and Y.