3 ms·
Yes, we block IPs that seem to be crawlers ignoring robots.txt. We've always blocked abusive IPs, but I tightened up the blocking a few weeks ago. A lot of peo
by pg 15y ago
Yes, we block IPs that seem to be crawlers ignoring robots.txt. We've always blocked abusive IPs, but I tightened up the blocking a few weeks ago. A lot of people were crawling HN, most of them unnecessarily because they were doing things they could have done more efficiently through HNSearch's API.
Some users may remember that the site had gotten really slow a few weeks ago. One of the reasons it's faster now is that we cracked down on crawlers.
- halostatue 15y agoThat makes sense. I suspect that the "cause" in this case was the chrome extension that I had been using (I don't remember the name offhand, but it's the one that shows changes in HN between visits). I usually left the tab open but only visited a couple of times a day, so it may have been doing something like that.
- ronnier 15y agoI think it's time I shut down http://ihackernews.com http://ihackernews.com. It's getting impossible to keep the IP address from being blocked. Really unfortunate, because a lot of people like to use the site.
- polyfractal 15y agoAnyone know if this what the WP7 HN App uses? I've noticed that the app no longer works which is a shame, I used it all the time.
- wx77 15y agoAppears so: https://github.com/jpf/hacker-news-wp7/blob/master/README https://github.com/jpf/hacker-news-wp7/blob/master/README If HN could give back json instead of html I'm guessing a lot of the crawling could be mitigated to client side. Kind of like appending .json to everything on reddit
- Timothee 15y agoI had to stop using it because of that. And I started to think of re-writing it as a self-hosted solution to hopefully avoid IP bans. I don't know if you'd consider releasing it for anyone to install on their own server, and if pg would be fine with that solution. (assuming that each page load would dynamically scrape a single page on ycombinator.com) edit: of course, it might still have its issues if a lot of people were to host it with the same host… Another solution would be for pg to add a mobile stylesheet. That's honestly the only reason why I would need it.
- pg 15y agoIf you tell me the ip addr I can whitelist it. pg@ycombinator.com
- ronnier 15y agoWow, thanks! Email sent.