3 ms·
You might want to tone down you scraping bot. Since posting http://news.ycombinator.com/item?id=1627619 http://news.ycombinator.com/item?id=1627619 your bot has
by dman 16y ago
You might want to tone down you scraping bot. Since posting http://news.ycombinator.com/item?id=1627619 http://news.ycombinator.com/item?id=1627619 your bot has hit us 100 times in the last 4 hours.
- e1ven 16y agoSorry about that! I didn't think it would be a problem, but I can see the programming error I made which is causing it. Normally, when we pull pages from RSS, we have a UUID attached to each, to avoid pulling it again. Since it's scraping HN, it doesn't get those, and isn't behaving nicely. The reason it pulls the destination page is to get the final link- Too many sites are tinyurl/bit.ly/etc, so I load them to see who I'm dealing with at the end of the day. I'll generate some sort of Token for those to avoid the problem later tonight. Thanks, and again, Sorry for the trouble!
- e1ven 16y agoI've added a new check to the code- It'll check the base URL against existing stories, before doing the URL normalization. This should fix the problem for every URL that isn't a tinyurl. Thanks for the heads-up. Can you verify if it's nicer for your server now?