4 ms·

Pierre, thanks for stopping by to confirm these issues. I often see sites be inconsistent between www and non-www, or between http and https. It looks like safe

Pierre, thanks for stopping by to confirm these issues. I often see sites be inconsistent between www and non-www, or between http and https. It looks like safeshepherd.com was doing both. More consistent redirects and adding rel=canonical should definitely help us figure out which url you prefer. Just to confirm what I said elsewhere, this site doesn't have any manual spam actions or anything like that. It's just a matter of Google trying to pick the correct canonical url when you have a lot of different (www, non-www, http, https) urls you're showing. If you make things more consistent, I think Google will stabilize on your preferred url pretty quickly.

jeebus 14y ago
Matt & Pierre, thanks for your thoughts and sorry that this ended up being a rookie mistake. I have a rel canonical good to go. Thanks again for your time.
- Matt_Cutts 14y ago
  No worries at all--glad it turned out to be easily fixable. And now I know what "nerfed" means. :)
  - kayge 14y ago
    FYI - Another fairly common usage of "nerfed" these days (especially in the gaming community) refers to something being toned-down. E.g. if many players are complaining about a character ability being too powerful, the developers may consider "nerfing" that character.
    - deleted 14y ago
      [deleted]
  - sixQuarks 14y ago
    This is why I love HackerNews. Guy asks for SEO help, frigg'n Matt Cutts answers!
  - matznerd 14y ago
    In Internet Marketing, we refer to being delisted as getting "sandboxed"
    - mh- 14y ago
      As you're replying to Google's Web Spam lead, I'm sure he's familiar with the terms for obliterating your splog farms. :)
- mjwalshe 14y ago
  I have also seen a messed up (404 erroring) robots.txt file cause a site to get deindexed out of the blue
  - pierrefar 14y ago
    That's a misconception. A 404 on robots.txt will not have any effect on crawling as it's treated the same as an empty robots.txt file allowing all crawling. But it's different for 5xx HTTP errors for the robots.txt file. As Googlebot is currently configured, it will halt all crawling of the site if the site’s robots.txt file returns a 5xx status code for robots.txt. This crawling block will continue until Googlebot sees an acceptable status code for robots.txt fetches (HTTP 200 or 404).
    - mjwalshe 14y ago
      interesting that needs to go into the webmaster guidelines I was not seeing 500's or having it reported in GWT as errors on the site that it happened to
      - pierrefar 14y ago
        It doesn't belong in the guidelines but it is described in the relevant section of the Help Center: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=35147 http://support.google.com/webmasters/bin/answer.py?hl=en&#38... In summary: If for any reason we cannot reach the robots.txt due to an error (e.g a firewall blocking Googlebot or a 5xx error code when fetching) Googlebot stops its crawling and it's reported in Webmaster Tools as a crawl error. That Help Center article above is about the error message shown in Webmaster Tools. Given that you said you did not see errors being reported, That suggests there was something else going on. If you need more help, our forums are a great place to ask.
        mjwalshe 14y ago
        Chears I am of on leave for a week ill get this put into our best practice guide for our devs and IS guys when I am back. Funny thing was I tried resubmitting the main page in GWT an all the traffic came back almost instantly.
coryj 14y ago
Do we still have to do the redirects if we have this setup properly in webmaster tools? That is, under configuration --> settings --? preferred domain. Second, in webmaster tools, should we always have the www and non-www setup so we can do the "change of address". For example, if www.mysite.com is my preferred URL, do I need to also make sure mysite.com is in webmaster tools and change the address to go to www.mysite.com?
- robdwoods 14y ago
  My SOP is to make sure first that all non-canonical versions of the home page (the page with the most value, usually) are redirected, then make sure you set your preferred address in WMT, then add the rel=canonical tag to capture all the possible versions of the home page that you can't think of. As a side note I also noticed (the hard way) that Google treats capitalization in URIs the same way as these examples so www.Example.com is treated differently (at least for link value) than www.example.com. Basically if there is a single different character in the URI then it's considered a different URI.
robomartin 14y ago
I have a question: Why are these considered different sites by your algos? If we were talking ".com" vs ".net", OK, I get it. But this is about "www.domain.com" vs "domain.com" and their http and https variants. I'm sure there's something I don't understand. Would "http:www.apple.com", "https:www.apple.com", "http:apple.com" and "https:apple.com" be treated by Google as four completely different and separate sites also to be ranked in isolation of each other? Why?
- jlarocco 14y ago
  "www" might be a very special case, but there are lots of times where "this.domain.com" is completely unrelated to "that.domain.com". Many sites, for example give users their own "name.whatever.com" subdomain. In those cases treating the sites as the same doesn't make any sense.
  - superchink 14y ago
    True, but I think that is just the special case being asked about. Would it make sense to have an exception for “www”?
  - robomartin 14y ago
    That's true, but the www/no-www/http/https cases are very likely to refer to exactly the same site. Besides, Google's algos, through crawling, should know that it is the same site. It seems unfair to punish the site for this.

jeebus 14y ago

Matt & Pierre, thanks for your thoughts and sorry that this ended up being a rookie mistake. I have a rel canonical good to go. Thanks again for your time.

Matt_Cutts 14y ago
No worries at all--glad it turned out to be easily fixable. And now I know what "nerfed" means. :)
- kayge 14y ago
  FYI - Another fairly common usage of "nerfed" these days (especially in the gaming community) refers to something being toned-down. E.g. if many players are complaining about a character ability being too powerful, the developers may consider "nerfing" that character.
  - deleted 14y ago
    [deleted]
- sixQuarks 14y ago
  This is why I love HackerNews. Guy asks for SEO help, frigg'n Matt Cutts answers!
- matznerd 14y ago
  In Internet Marketing, we refer to being delisted as getting "sandboxed"
  - mh- 14y ago
    As you're replying to Google's Web Spam lead, I'm sure he's familiar with the terms for obliterating your splog farms. :)
mjwalshe 14y ago
I have also seen a messed up (404 erroring) robots.txt file cause a site to get deindexed out of the blue
- pierrefar 14y ago
  That's a misconception. A 404 on robots.txt will not have any effect on crawling as it's treated the same as an empty robots.txt file allowing all crawling. But it's different for 5xx HTTP errors for the robots.txt file. As Googlebot is currently configured, it will halt all crawling of the site if the site’s robots.txt file returns a 5xx status code for robots.txt. This crawling block will continue until Googlebot sees an acceptable status code for robots.txt fetches (HTTP 200 or 404).
  - mjwalshe 14y ago
    interesting that needs to go into the webmaster guidelines I was not seeing 500's or having it reported in GWT as errors on the site that it happened to
    - pierrefar 14y ago
      It doesn't belong in the guidelines but it is described in the relevant section of the Help Center: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=35147 http://support.google.com/webmasters/bin/answer.py?hl=en&#38... In summary: If for any reason we cannot reach the robots.txt due to an error (e.g a firewall blocking Googlebot or a 5xx error code when fetching) Googlebot stops its crawling and it's reported in Webmaster Tools as a crawl error. That Help Center article above is about the error message shown in Webmaster Tools. Given that you said you did not see errors being reported, That suggests there was something else going on. If you need more help, our forums are a great place to ask.
      - mjwalshe 14y ago
        Chears I am of on leave for a week ill get this put into our best practice guide for our devs and IS guys when I am back. Funny thing was I tried resubmitting the main page in GWT an all the traffic came back almost instantly.

coryj 14y ago

Do we still have to do the redirects if we have this setup properly in webmaster tools? That is, under configuration --> settings --? preferred domain. Second, in webmaster tools, should we always have the www and non-www setup so we can do the "change of address". For example, if www.mysite.com is my preferred URL, do I need to also make sure mysite.com is in webmaster tools and change the address to go to www.mysite.com?

robdwoods 14y ago
My SOP is to make sure first that all non-canonical versions of the home page (the page with the most value, usually) are redirected, then make sure you set your preferred address in WMT, then add the rel=canonical tag to capture all the possible versions of the home page that you can't think of. As a side note I also noticed (the hard way) that Google treats capitalization in URIs the same way as these examples so www.Example.com is treated differently (at least for link value) than www.example.com. Basically if there is a single different character in the URI then it's considered a different URI.

robomartin 14y ago

I have a question: Why are these considered different sites by your algos? If we were talking ".com" vs ".net", OK, I get it. But this is about "www.domain.com" vs "domain.com" and their http and https variants. I'm sure there's something I don't understand. Would "http:www.apple.com", "https:www.apple.com", "http:apple.com" and "https:apple.com" be treated by Google as four completely different and separate sites also to be ranked in isolation of each other? Why?

jlarocco 14y ago
"www" might be a very special case, but there are lots of times where "this.domain.com" is completely unrelated to "that.domain.com". Many sites, for example give users their own "name.whatever.com" subdomain. In those cases treating the sites as the same doesn't make any sense.
- superchink 14y ago
  True, but I think that is just the special case being asked about. Would it make sense to have an exception for “www”?
- robomartin 14y ago
  That's true, but the www/no-www/http/https cases are very likely to refer to exactly the same site. Besides, Google's algos, through crawling, should know that it is the same site. It seems unfair to punish the site for this.