12 ms·
I think I know what the problem is; we're detecting HN as a dead page. It's unclear whether this happened on the HN side or on Google's side, but I'm pinging th
by Matt_Cutts 15y ago
I think I know what the problem is; we're detecting HN as a dead page. It's unclear whether this happened on the HN side or on Google's side, but I'm pinging the right people to ask whether we can get this fixed pretty quickly.
Added: Looks like HN has been blocking Googlebot, so our automated systems started to think that HN was dead. I dropped an email to PG to ask what he'd like us to do.
- deleted 15y ago[deleted]
- moe 15y agoDoes it perhaps return something to the tune of "Unknown or expired link"? :-P
- Matt_Cutts 15y agoNope, it looks like we haven't been able to fetch even the robots.txt file for a few days, so our system is trying to play it safe by assuming that HN is unreachable/dead.
- asto 15y agoWhoooosh!
- pg 15y agoI sent you an email about this. (A couple weeks ago I banned all Google crawler IPs except one. Crawlers are disproportionately bad for HN's performance because HN is optimized to serve recent stuff, which is usually in memory.)
- tibbon 15y agoCould you use some sort of sitemap or other way to provide the data to Google that isn't so damaging to site performance? Or in Google Webmaster tools turn down the rate of crawling? Just realized that this could be a problem for lots of sites, and I'm curious as to what the best solution is, since not everyone has Matt Cutts reading their site and helping out.
- Matt_Cutts 15y agoWe do have a self-service tool in webmaster tools for people who prefer to be crawled slower.
- arctangent 15y agoDo you support/respect the "crawl-delay" directive? http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl...
- storborg 15y agoNope. As mentioned above, apparently Google thinks people will "shoot themselves in the foot" with the crawl-delay directive, while they won't with Google's special interface (which requires registering and logging in).
- wanorris 15y agoI can't imagine that they are just guessing abut this. I'm sure someone tried implementing it and was horrified at the actual results before they gave up on it.
- einhverfr 15y agoWhat more can you expect from the World's Largest Adware Company? Seriously, one thing about Google is that they seem to really like ensuring people are logged on, preferably at all times. Fortunately recent changes to Google Apps (promoting apps user accounts to full Google accounts) has made this more complex on my side and probably degraded the level of actionable info they can get out of it.
- Thieum22 15y agoor faster : http://news.ycombinator.com/item?id=2382728 http://news.ycombinator.com/item?id=2382728
- pierrefar 15y agoHi Paul, A site can be crawled from any number of Googlebot IP addresses, and so blocking all except one doesn't help in throttling crawling. If you verify the site in Webmaster Tools, we have a tool you can use to set a slower crawl rate for Googlebot, regardless of which specific IP address ends up crawling the site. Let me know if you need more help. Edit Detailed instructions to set a custom crawl rate: 1. Verify the site in Webmaster Tools. 2. On the site's dashboard, the left hand side menu has an entry called Site Settings. Expand that and choose the Settings submenu. 3. The page there has a crawl rate setting (last one). It defaults to " Let Google determine my crawl rate (recommended)". Select "Set custom crawl rate" instead. 4. That opens up a form and choose his desired crawl rate in crawls per second. If there is a specific problem with Googlebot, you can reach the team as follows: 1. To the right hand side of the Crawl Rate setting is a link called "Learn More". Click that to open a yellow box. 2. In the box is a link called Report a problem with Googlebot which will take you to form you can fill out with full details. Thanks! Pierre
- aw3c2 15y agoI would like to set that crawl rate but do not see why I must register at Google to do so. Why can't Google support the Crawl-Delay directive in robots.txt for this?
- Matt_Cutts 15y agoMy plane is about to take off, but very briefly: people sometimes shoot themselves in the foot and get it way, way wrong. Like "crawl a single page from my website every five years" wrong. Crawl-Delay is (in my opinion) not the best measure. We tend to talk about "hostload," which is the inverse: the number of simultaneous connections that are allowed.
- anonfoobar1 15y agoAnother great way to shoot yourself in the foot (and getting it way way wrong) is to block all Googlebot IPs except for one.
- 15y ago
- Matt_Cutts 15y agoGotcha--thanks, Paul. I'm about to get on a plane, but we'll get this figured out where we're not sending as much hostload toward HN.
- ashishgandhi 15y agoDoes that mean at Google you can manually set the system to treat different sites differently? I don't mean the ranking but other aspects - like you guys blacklisted some domains which produce low quality content in wholesale. (I don't know if the algorithm was tweaked to detect and filter such sources or if it was a manual thing.)
- Kiall 15y agoNo, It sounds like @pg has blocked google bot via a firewall, So as far as google's automated systems are concerned, hacker news was shutdown and thus removed from the index. I believe Matt is simply offering help/advice on figuring out the best solution for HN.
- seabee 15y agoDoes that mean at Google you can manually set the system to treat different sites differently? Webmaster Tools has a crawl rate slider which operate on a site-by-site basis, and that's existed for quite a while now. If you're asking if they can manually boost a site's ranking, hopefully that isn't what's being suggested.
- wgx 15y agoIf only we could all get our Google woes fixed in such a manner.
- metaprinter 15y agoseriously. i can't get a hospital to show up in google maps... no human for me to talk to. HN is number 4 instead of 1 google page one, they're right on it.
- wheels 15y agoHave you considered putting a caching reverse proxy in front of the arc app to keep the backend from having to render all of the old pages? It seems like the only dynamic element of old articles is the "$x days ago" bit and that'd be pretty easy to turn into something static by instead just putting in timestamps in the actual HTML and using Javascript to transform them into how many hours / days ago they were. Then the crawlers would just be pulling out cached, pre-rendered HTML. There's an example of doing such with nginx here: http://serverfault.com/questions/30705/how-to-set-up-nginx-as-a-caching-reverse-proxy http://serverfault.com/questions/30705/how-to-set-up-nginx-a... With that you'd just have to send out the HTTP header from the arc app saying that current articles expire immediately, and old ones don't.
- pg 15y agoI believe Rtm has already set one up.
- wheels 15y agoThe conspicuous lack of a "Server:" header inclines me to believe that that's probably not the case (most web servers set one indicating the server software and version). Here are the headers that HN sends out from an old post (20 days ago): HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 Cache-Control: private Connection: close Cache-Control: max-age=0
- bascule 15y agoMy favorite part of HN's headers: the lines are separated by naked LFs instead of CRLF, in violation of the HTTP spec
- divtxt 15y agoThis is common violation that everyone accepts. It's definitely done by 'bad' clients - not sure how often servers send bare LF. (I used to telnet to port 80 for testing, and type GET / HTTP/1.0 <enter> <enter>, and that should be LF on Linux & Mac)
- jedberg 15y agoreddit had the same problem. We set up a separate server just for the google crawler with it's own copy of the database, so that the queries for old pages didn't slow down everyone else.
- RBerenguel 15y agoAdvantages of having a deep Google guy as a user :)
- cyanbane 15y agoThe question is, 'Was Matt Cutts ever THE Googleguy?' ( http://www.webmasterworld.com/profilev4.cgi?action=view&member=GoogleGuy ) I always wondered who it really was years back.
- Matt_Cutts 15y agoYes, I'm GoogleGuy. Steven Levy documented that in his recent "In The Plex" book, so there's no harm in verifying it now.
- ComputerGuru 15y agoMatt, when you have a free minute, can you take a look at this? http://news.ycombinator.com/item?id=3272528 http://news.ycombinator.com/item?id=3272528
- Matt_Cutts 15y agoSure, some indexing folks are discussing this now.
- ComputerGuru 15y agoI appreciate it. I set up a 301 from dl.php?id=x to /myproduct/ for each of the ids last night as a last-ditch effort.
- ComputerGuru 15y agobtw, turns out the redirect I set up two days had broken a couple of links, but not the ones I care about. Fixed now, though.
- blhack 15y agoSeeing threads like this remind me that HN is still a pretty tight-knit community of real people doing real things. It's good to see this stuff sometimes. Thanks, Matt! edit: And then reading some of the other threads on this topic is a bit...something. Guys, can you calm the conspiracy theory nonsense a bit? Please? If you're not on this site very much, you might not realize that Matt pops into almost every thread where google is doing something strange regardless of who they're doing it to, and tries to help figure out what is happening. This isn't HN getting some sort of preferential treatment, this is just the effect of having a userbase full of hackers. You'd see the same type of thing on /. years ago if you frequented it enough. This is nothing new. This is what a good community looks like. Everybody relax. Honestly if you read the things that Matt and Pierre have said, they just looked at "freshness" (I believe that is what it is called), and inferred that PG had blocked their crawlers. This is all stuff you can get from within google webmaster tools (which isn't some secret whoooo insider google thing. It's something they offer to everybody, and it's just like analytics.) OH! Wait! I mean (hold on, let me spin up my google conspiracy theory generator): thehackernews.com has more ads on it so google is intentionally tweaking their algo to serve that page at a higher point than the real HN because of ads! DUH! C'mon, guys, look at their user pages. They're both just active users of the site trying to help out.
- billpatrianakos 15y agoIt really is but I was personally kind of glad not to see HN have that high of a ranking on Google for the term. I remember when I first started visiting HN I saw all these smart people and the tight community and I was amazed that something that felt so close-knit and exclusive yet was still open could still exist these days. I was a lurker for a long time before I actually signed up and participated because I honestly felt like I swasnt entitled to be part of "the group" and I should somehow earn my wings. Then in late 2010 I signed up but didn't submit for a bit and didn't join discussions. I still felt like I didn't have enough to offer. I now feel like I've somehow earned the right to be part of this community though in hindsight I'm quite embarrassed of my first few submissions. So this story does have a point that I'm about to get to. I first heard of HN through an article in GQ and then forgot the link. I couldn't find the site again after searching Google for "Hacker News" as easily as I thought. This frustrated me slightly back then but now I think it's a good thing. As the size of a community gets larger the quality of comments and submissions usually decreases. Letting people join HN freely and openly is a great thing but I fear that if it became a huge sensation then we'd be inundated by garbage submissions and comments way more frequently. I know about the post on how newbies often say HN is becoming Reddit and all that so I do try to remember that. So the point is that not everyone respects communities like this and are thoughtful about joining and how they choose to interact on communities like HN the same way I was and I feel like maybe it's okay if Google isn't giving us the best ranking for certain terms. I mean, HN is easy to find still, just not that easy to stumble over.
- GigabyteCoin 15y agoApparently the page in the top spot now hasn't been updated since: 2011.07.08 It now only consists of ads, a twitter feed, and a "Abba-da-dabba-da-dabba-dabba Dat’s all folks!" line. So what if Googlebot thought HN was dead? Why would it opt to show a "more dead" page in place of it? I think you're fibbing.
- StavrosK 15y agoThe other page isn't dead, they can access it.
- eurohacker 15y agoyes - it looks more like google had been changing their algorithm recently - in favor of older sites , versus the content - and it didnt turn out quite well ...
- eurohacker 15y agocan you improve my site also Matt and fix the issue , i had the same problem some time ago - some wordpress plugin blocked googlebot unintentinally and i lost the ranking, i am sure here are many folks here as well who had similar problems, why not help us all out ))
- jfoster 15y agoHas Google considered an indicator next to results for this type of case? A single red exclamation mark with further details when the mouse is over it, for example. I'm aware of webmaster tools, but it seems not all webmasters are.
- Matt_Cutts 15y agoWe do include some information in the search results for cases where our users could be harmed, e.g. if we detect malware or if we think that your site may have been hacked. We've been exploring other ways to alert site owners that there's something important affecting their site. But people do choose to remove their sites, so we can't always tell between a mistake vs. someone who genuinely prefers not to be in Google's index.
- benatkin 15y agoI never thought that Google Search would be so laughable.
- deleted 15y ago[deleted]
- billpatrianakos 15y agoExcellent contribution... Ugh, this is why I don't mind if HN is listed on Google or not. So which part is laughable? The part where they're able to crawl the vast expanses of the web and return relevant results for the majority of their users? Or is it the part where they came out of nowhere to dominate search because they did it better than the rest? Come on now, you can't be all things to all people. Google is far from perfect but for a lot of us it's much closer to perfect than the competition and they're constantly trying to improve it. Why don't you go ask Matt Cutts to fix whichever parts of it you think are laughable to your liking? He's been hanging around here and he doesn't seem shy about answering people's questions and concerns. I do doubt he'd give the time of day to a one sentence remark that adds nothing of value whatsoever to the larger discussion or any of it's offshoots.
- deleted 15y ago[deleted]
- benatkin 15y agoNo, the part where instead of getting better, Google search keeps getting worse. The verbatim mode is finally there but much of the time when it mucks with my search terms it's wrong. The UI is also cluttered. They are doing a lot of things that do nothing to improve the quality of the search, so when I want to learn something, I often skip Google and go straight to Stack Overflow or Wikipedia. I did jump the gun on this issue, though...turns out some googlebot blocking was involved.
- mutt_cutts 15y agowhat about the massive amount of duplicate content indexed via this domain: hackerne.ws ?