3 ms·
I use this, but it can be a little annoying now that some sites like StackExchange block EC2 netblocks (I guess because they have lots of juicy content ripe for
by ben1040 15y ago
I use this, but it can be a little annoying now that some sites like StackExchange block EC2 netblocks (I guess because they have lots of juicy content ripe for scraping). Yelp is another one that comes to mind that does this as well.
- ldar15 15y agoWhat??? For real? Fuck. I was going to set up an EC2 vpn endpoint this very day.
- ben1040 15y agoYep. Some more discussion here: http://news.ycombinator.com/item?id=2441535 http://news.ycombinator.com/item?id=2441535 It sucks, but I can see the reason why they just blanket block EC2 rather than trying to find any more nuanced way to identify scrapers. If I were running scraping bots on EC2, rate-limiting me isn't a real threat since the instant I get throttled for misuse I can just destroy the instance, start up a brand new one on a different IP address, and continue as I was before.