3 ms·
Couldn't they just rate limit them? Are they literally using a new IP for every request?
by saghm 4mo ago
Couldn't they just rate limit them? Are they literally using a new IP for every request?
- anematode 4mo agoThere are indeed scrapers which use tens of thousands of distinct IPs, and so rate limiting them isn't a solution.
- sroussey 4mo agoWe used to call this snow shoeing.
- kstrauser 4mo agoIn my experience, yes. At the peak of a scraper flood I was dealing with, I'd say about 90% of the traffic was from a unique IP. I'd never seen anything quite like that before.