4 ms·
This person used a new IP address for every single request, so that won’t work. And that’s a growing trend.
by zenexer 6y ago
This person used a new IP address for every single request, so that won’t work. And that’s a growing trend.
- folkhack 6y agoYep I can come at someone with datacenter/residential/mobile/etc. IP addresses, all incredibly configurable to slip around network blocks. Luminati and Proxy Bonanza are the services I've used with the most success. Getting source proxy lists of high-reputation networks is just $$ and a simple API integration game anymore.
- zenexer 6y agoThere’s also the issue of CGNAT. If you rate limit too strictly based on IP address, you harm users who are stuck with CGNAT, especially in Asia and Africa. India is particularly problematic. As for stuff like Luminati, if you’re being sufficiently sneaky, chances are you’re not going to snowball in the first place. I’m not sure why anyone would bother paying for Luminati to crawl sites like the one for which I work, but I have seen people use it to scam. We can’t really be bothered to waste resources blocking well-behaved crawlers. Just keep it at a reasonable pace, respect errors (especially 429, but also 410 and 503), and ensure we have a way to contact you if things go wrong.
- folkhack 6y agoYep - I have the HTTP error code detection dialed into an extreme because it's dumb to run a broken scrape anyway. Frankly just any errors - if I see more than say 5-10 jobs fail within a 2-3 minute time period things are designed to wait X time, try again... and stop if they're still encountering errors and ping me to come in and investigate. Faulty retry logic is just as dangerous as the forked/distributed run-off situation.
- CodesInChaos 6y agoI'd assume they use 1000 distinct IPs because the system scaled to 1000 instances and presumably many of them came from related IPs. So it makes IP banning considerably harder, but not impossible.
- zenexer 6y agoMore importantly, any reasonable IP-based approach would have a lot of false positives. What if there’s an RSS service like Feedly running on the same cloud?
- CodesInChaos 6y agoDepending on your product, blacklisting all AWS IPs might be acceptable. For example my company has a VPN exit on AWS, which appears to be blacklisted by twitter.