4 ms·

HN Banned from Google

8 points by jayjay1010 13y ago

pg 13y ago
I'm sure it's just automatic. Their sw thinks HN is dead because it stops giving their crawlers pages after a while.
- lnanek2 13y ago
  Ah, that would makes sense. Happens to me all the time when I leave HN pages open for a while then try to use them later. I imagine the bots try to follow links they picked up earlier a long time later as well at times. Kind of funny a bad session system on a site can kill your search rankings, though. It would be fatal for sites that live off Google traffic, and tough for people using off the shelf software to deal with.
- jayjay1010 13y ago
  It's NEVER been an issue before this time with consistent top results for years and this suggests otherwise https://www.google.co.uk/search?q=site:news.ycombinator.com&start=220&client=safari&sa=N&hl=en-gb&nomo=1&biw=320&bih=416 https://www.google.co.uk/search?q=site:news.ycombinator.com&... Btw this is showing the pages indexed for the site HN. That its not showing in the results now for your brand WILL have devastating results on your traffic directly from Google. Though you might not care?
- jayjay1010 13y ago
  Specifically your no longer ranking for your main brand! That's nothing to do with your long tail search terms.
  - tptacek 13y ago
    Whoah, you're right! If you search for [Hacker News], you get a bunch of junk results now. That's so cool!
muuck 13y ago
His Hacker News query give me https://news.ycombinator.com/jobs https://news.ycombinator.com/jobs back as first result. Just for fun I tried the same query on DDG and the 27th result was the first ycombinator domain I got back.
- lnanek2 13y ago
  Maybe jobs have non-session/continuation based perma-links? That would allow them to work even for things like the Google Crawler.
cddotdotslash 13y ago
I clicked the links and every link on the frontpage is to different Hacker News pages. I see nothing wrong here.
- jayjay1010 13y ago
  It's wrong bc it's NOT bringing up the best result for the query.
  - greenyoda 13y ago
    I'm getting the welcome page for HN as my third result on the "hacker news" (without the quotes) search: http://ycombinator.com/newswelcome.html http://ycombinator.com/newswelcome.html This is arguably the best result for the query, since it explains what HN is about and provides a link to the front page. (People who have already used HN know where it is and don't need to search for it.) I block cookies from Google, so they're not relying on my search history to give me relevant results. And I'm not sure why searching for "HN" would be reasonably expected to get someone to Hacker News. After all, "hn" is the top-level domain for Honduras, and could stand for a million other things as well. It's probably accidental that it ever worked at all.
  - cddotdotslash 13y ago
    And what is the "best" result in your opinion? I'm getting the homepage of Hacker News (news.ycombinator) along with a nice description. I'm as skeptical as the next guy when it comes to this kind of stuff, but this is just plain ridiculous.
nbpoole 13y ago
Luckily, this question has been asked and answered fairly recently! Specifically https://news.ycombinator.com/item?id=5955043 https://news.ycombinator.com/item?id=5955043 from last week, complete with a response from Matt Cutts (https://news.ycombinator.com/item?id=5955374 https://news.ycombinator.com/item?id=5955374): > It's not that PG has a grudge against Google (or vice versa) or anything like that. I believe that search engine bots crawl Hacker News hard enough that PG blocks most crawling by bots. In the case of Google, he does allow us to crawl from some IP addresses, but it's true that Google isn't able to crawl/index every page on Hacker News. > Here's a link where I answered the same question about three weeks ago: https://news.ycombinator.com/item?id=5837004 https://news.ycombinator.com/item?id=5837004 , so this isn't a new issue. In fact, PG has been blocking various bots since 2011 or so; https://news.ycombinator.com/item?id=3277661 https://news.ycombinator.com/item?id=3277661 is one of the original discussions about this. > And to show this isn't a Google-specific issue, note that Bing's #1 result for the search [hacker news] is a completely different site, thehackernews.com: http://www.bing.com/search?q=hacker+news http://www.bing.com/search?q=hacker+news > In general, I think PG's priority is to have a useful, interesting site for hackers. That takes precedence and is the reason why I believe PG blocks most bots: so that crawling doesn't overload the site.
- jayjay1010 13y ago
  Well I can't no longer find the site by searching for the brand anymore and I'm sure this will dramatically change the traffic the site gets.
  - thaumaturgy 13y ago
    Good.
- cgjung 13y ago
  I'm not buying the answer Matt gave. HN was "banned" because of the anti-google comments expressed on this site. Google doesn't want other people (who are unaware google is selling their privacy to the NSA) to accidentally stumble across the comments on HN.

8 points by jayjay1010 13y ago

pg 13y ago

I'm sure it's just automatic. Their sw thinks HN is dead because it stops giving their crawlers pages after a while.

lnanek2 13y ago
Ah, that would makes sense. Happens to me all the time when I leave HN pages open for a while then try to use them later. I imagine the bots try to follow links they picked up earlier a long time later as well at times. Kind of funny a bad session system on a site can kill your search rankings, though. It would be fatal for sites that live off Google traffic, and tough for people using off the shelf software to deal with.
jayjay1010 13y ago
It's NEVER been an issue before this time with consistent top results for years and this suggests otherwise https://www.google.co.uk/search?q=site:news.ycombinator.com&start=220&client=safari&sa=N&hl=en-gb&nomo=1&biw=320&bih=416 https://www.google.co.uk/search?q=site:news.ycombinator.com&... Btw this is showing the pages indexed for the site HN. That its not showing in the results now for your brand WILL have devastating results on your traffic directly from Google. Though you might not care?
jayjay1010 13y ago
Specifically your no longer ranking for your main brand! That's nothing to do with your long tail search terms.
- tptacek 13y ago
  Whoah, you're right! If you search for [Hacker News], you get a bunch of junk results now. That's so cool!

muuck 13y ago

His Hacker News query give me https://news.ycombinator.com/jobs https://news.ycombinator.com/jobs back as first result. Just for fun I tried the same query on DDG and the 27th result was the first ycombinator domain I got back.

lnanek2 13y ago
Maybe jobs have non-session/continuation based perma-links? That would allow them to work even for things like the Google Crawler.

cddotdotslash 13y ago

I clicked the links and every link on the frontpage is to different Hacker News pages. I see nothing wrong here.

jayjay1010 13y ago
It's wrong bc it's NOT bringing up the best result for the query.
- greenyoda 13y ago
  I'm getting the welcome page for HN as my third result on the "hacker news" (without the quotes) search: http://ycombinator.com/newswelcome.html http://ycombinator.com/newswelcome.html This is arguably the best result for the query, since it explains what HN is about and provides a link to the front page. (People who have already used HN know where it is and don't need to search for it.) I block cookies from Google, so they're not relying on my search history to give me relevant results. And I'm not sure why searching for "HN" would be reasonably expected to get someone to Hacker News. After all, "hn" is the top-level domain for Honduras, and could stand for a million other things as well. It's probably accidental that it ever worked at all.
- cddotdotslash 13y ago
  And what is the "best" result in your opinion? I'm getting the homepage of Hacker News (news.ycombinator) along with a nice description. I'm as skeptical as the next guy when it comes to this kind of stuff, but this is just plain ridiculous.

nbpoole 13y ago

Luckily, this question has been asked and answered fairly recently! Specifically https://news.ycombinator.com/item?id=5955043 https://news.ycombinator.com/item?id=5955043 from last week, complete with a response from Matt Cutts (https://news.ycombinator.com/item?id=5955374 https://news.ycombinator.com/item?id=5955374): > It's not that PG has a grudge against Google (or vice versa) or anything like that. I believe that search engine bots crawl Hacker News hard enough that PG blocks most crawling by bots. In the case of Google, he does allow us to crawl from some IP addresses, but it's true that Google isn't able to crawl/index every page on Hacker News. > Here's a link where I answered the same question about three weeks ago: https://news.ycombinator.com/item?id=5837004 https://news.ycombinator.com/item?id=5837004 , so this isn't a new issue. In fact, PG has been blocking various bots since 2011 or so; https://news.ycombinator.com/item?id=3277661 https://news.ycombinator.com/item?id=3277661 is one of the original discussions about this. > And to show this isn't a Google-specific issue, note that Bing's #1 result for the search [hacker news] is a completely different site, thehackernews.com: http://www.bing.com/search?q=hacker+news http://www.bing.com/search?q=hacker+news > In general, I think PG's priority is to have a useful, interesting site for hackers. That takes precedence and is the reason why I believe PG blocks most bots: so that crawling doesn't overload the site.

jayjay1010 13y ago
Well I can't no longer find the site by searching for the brand anymore and I'm sure this will dramatically change the traffic the site gets.
- thaumaturgy 13y ago
  Good.
cgjung 13y ago
I'm not buying the answer Matt gave. HN was "banned" because of the anti-google comments expressed on this site. Google doesn't want other people (who are unaware google is selling their privacy to the NSA) to accidentally stumble across the comments on HN.