4 ms·
Don't worry, it doesn't mean anything. The software for ranking applications runs on the same server, and it is horribly inefficient (something 4 people use ev
by pg 17y ago
Don't worry, it doesn't mean anything. The software for ranking applications runs on the same server, and it is horribly inefficient (something 4 people use every 6 months doesn't tend to get optimized much). This weekend all of us were reading applications at the same time, and the system was getting so slow that I banned crawlers for a bit to buy us some margin. (Traffic from crawlers is much more expensive for us than traffic from human users, because it interacts badly with lazy item loading.) We only finished reading applications an hour before I had to leave for SXSW, so I forgot to set robots.txt back to the normal one, but I just did now.
- g0atbutt 17y agoThat makes sense. Thanks for the explanation pg!
- lisper 17y agoDude, why not just get yourself another server?
- sbhat7 17y ago"something 4 people use every 6 months doesn't tend to get optimized much"
- _delirium 17y agoYeah, but the usual case is still to throw stuff like that on a spare "random odds and ends go here" server, not a production one whose performance/responsiveness you care about.
- deleted 17y ago[deleted]
- kmt 17y agoIt likely has a dependency on HN: looking up users, karma, etc., which is just simpler to do on the same machine.
- markbao 17y agoRanking applications by HN karma? I hope not.
- davidu 17y agoThey've been clear that they do look at your HN posts and comments, etc. It may or may not be used for ranking in specific, but it's definitely used and they've always been upfront about that when asked.
- idoh 17y agoAlso, they use hacker news to communicate to applicants. I applied a while back, and one day while browsing I got a message through HN, something like "pg has a question for you" in the header bar. Clicking through is the question, and place to put the answer. Basically, there's a whole messaging system in hacker news that most people don't see.
- markbao 17y agoYeah, this is known :) but an algo that incorporates HN karma wouldn't be the best thing :P
- Xichekolas 17y agoI was told once that they don't rank on karma, but if they recognize your name from discussions around here it helps your case.
- snprbob86 17y agoI'd be really worried about some algorithm somewhere deciding to crawl less frequently even after the temporary ban has been lifted.
- jsrfded 17y agoRobot bans can render your archive.org records inaccessable, zap all your former code 200 urls out of the major search indices, and yes, kill your inclusion in the list of topical sites. Much better to set a temporary Crawl-Delay directive. Otherwise you're not just telling the engines to pause crawling you, you're telling them "take all of my pages out of your index."
- Andys 17y agoWhat are the details of crawl-delay? If its supposed to be "number of seconds between hits", for some sites, 1 hit per second may as well be "not crawled at all"..
- ComputerGuru 17y agoRight, but his point is that at least this way you won't get delisted :)
- robryan 17y agoOn a related note the threads page seems to load very slowly now. I guess because it would place heavy load on the DB grabbing all your comments from different threads.
- duck 17y agoWhy is this still on the front page of HN? This seems like such a non-story.
- ks 17y agoHave you considered using last-modified/etag headers to control the crawlers? If you detect a crawler, and it requests a page that it has visited before, you could just return a 304 without doing anything else.