4 ms·
At work we have a scraper that likes to use a particularly expensive search query using hundreds of different ips dozens of times per second, all so that they c
by learc83 8y ago
At work we have a scraper that likes to use a particularly expensive search query using hundreds of different ips dozens of times per second, all so that they can scrape data that is freely available from us as an XML feed at a different url.
Every time I found a way to fingerprint and block them, they'd change their bot to avoid detection. Captcha for rate limiting seemed like the least bad option.
However, eventually I decided that instead of blocking them, I'd return random results from the db. They weren't checking the data too thoroughly because to this day they haven't changed the bot to avoid my most recent pattern detection. They're still merrily scraping useless data.
- seotut2 8y agoRate limiting can be achieved without actually inconveniencing regular users. Put a exponentially growing delay on the server's response for requests coming in too quick succession.
- learc83 8y agoThe problem was that with the sheer number of ip addresses they were using, and with the rate that some normal users used that particular endpoint, regular users would have been inconvenienced--either by being forced to wait or by being forced to do a captcha. I would have set it to only show the captcha if the delay was active, so users would effectively have had a choice of wait or captcha.
- herpderp3dtwerp 8y agoAwesome idea giving them crappy data. I do the same thing with loyalty programs. Someone has already registered XXX-555-1234 as a phone number in your area code. I like to think I'm adding a little bit of chaos to their data tracking.
- robk 8y agoWhy not add a "PLEASE USE THE XML DUMMY!" line to the returned results when you figure out it's them?
- learc83 8y agoTried that. They didn't seem to care.