6 ms·

Hi I work at Google helping webmasters like this. As far as I can tell, there are a couple of interacting issues we're seeing on the site that can be causing

Hi I work at Google helping webmasters like this. As far as I can tell, there are a couple of interacting issues we're seeing on the site that can be causing what you're seeing. It's a bit technical, but it's easy for you to implement a fix. Firstly, our algorithms recently have been picking one of the following URLs as the canonical URL for the homepage: http://www.safeshepherd.com/ http://www.safeshepherd.com/ https://www.safeshepherd.com/ https://www.safeshepherd.com/ https://safeshepherd.com/ https://safeshepherd.com/ For example, I see that the non-HTTPS pages redirect to the HTTPS pages (e.g. http://www.safeshepherd.com/ http://www.safeshepherd.com/ to https://www.safeshepherd.com/ https://www.safeshepherd.com/), but the non-www pages do not redirect to the www pages (both https://www.safeshepherd.com/ https://www.safeshepherd.com/ and https://safeshepherd.com/ https://safeshepherd.com/ return content). When we find the same content on multiple URLs like this, our algorithms pick one representative URL, and over the past few weeks the choice has been changing. As of 3 days ago, the current choice is https://safeshepherd.com/ https://safeshepherd.com/ . As it stands, our algorithms are trying to figure out the right canonical URL format, but it's difficult in this kind of situation. You can help by redirecting to your preferred URL format (say https://www.safeshepherd.com/* https://www.safeshepherd.com/*), and our systems will pick up this signal, and that will be reflected in the search results and reporting in Webmaster Tools. Secondly, Webmaster Tools treats these as different sites. For example, you would need to verify and check the statistics of both https://www.safeshepherd.com/ https://www.safeshepherd.com/ and https://safeshepherd.com/ https://safeshepherd.com/ (as well as the HTTP versions) as they're separate sites. It may be that you're checking (say) the stats for http://www.safeshepherd.com/ http://www.safeshepherd.com/ but if our algos have picked the https://www.safeshepherd.com/ https://www.safeshepherd.com/ URLs as canonical, the search queries of the former will suddenly be closer to zero but the latter will be a more accurate reflection of the site's traffic. Hope this helps, Pierre

Codhisattva 14y ago
Wow! Kudos to Google putting someone out in the wild answering these kinds of questions. Is there a normal place to ask for web/index help or does HN serve that purpose?
- Matt_Cutts 14y ago
  http://productforums.google.com/forum/#!forum/webmasters http://productforums.google.com/forum/#!forum/webmasters Those folks would have spotted these issues pretty quickly.
- pierrefar 14y ago
  The best places are our forums ( http://productforums.google.com/forum/m/#!forum/webmasters http://productforums.google.com/forum/m/#!forum/webmasters ) and our regular webmaster office hours ( http://sites.google.com/site/webmasterhelpforum/en/office-hours http://sites.google.com/site/webmasterhelpforum/en/office-ho... ) where a webmaster support Googler (me and a few others) have a hangout on Google+ that anyone can join and ask about their site. We do them in many languages and time zones to cover the world as much as possible. Obviously we can't be everywhere and we can't answer every question, but we try as much as possible to help when we can.
Matt_Cutts 14y ago
Pierre, thanks for stopping by to confirm these issues. I often see sites be inconsistent between www and non-www, or between http and https. It looks like safeshepherd.com was doing both. More consistent redirects and adding rel=canonical should definitely help us figure out which url you prefer. Just to confirm what I said elsewhere, this site doesn't have any manual spam actions or anything like that. It's just a matter of Google trying to pick the correct canonical url when you have a lot of different (www, non-www, http, https) urls you're showing. If you make things more consistent, I think Google will stabilize on your preferred url pretty quickly.
- jeebus 14y ago
  Matt & Pierre, thanks for your thoughts and sorry that this ended up being a rookie mistake. I have a rel canonical good to go. Thanks again for your time.
  - Matt_Cutts 14y ago
    No worries at all--glad it turned out to be easily fixable. And now I know what "nerfed" means. :)
    - kayge 14y ago
      FYI - Another fairly common usage of "nerfed" these days (especially in the gaming community) refers to something being toned-down. E.g. if many players are complaining about a character ability being too powerful, the developers may consider "nerfing" that character.
      - deleted 14y ago
        [deleted]
    - sixQuarks 14y ago
      This is why I love HackerNews. Guy asks for SEO help, frigg'n Matt Cutts answers!
    - 14y ago
harshreality 14y ago
Why doesn't google look at identical url paths on https, http, www, and no-www variants of the url, and if they look similar then use some default google policy to select which of them is canonical? For example, if http://mydomain.com/path http://mydomain.com/path and https://www.mydomain.com/path https://www.mydomain.com/path have 95% content correlation and repeated requests to http://mydomain.com/path http://mydomain.com/path have 95% content correlation, and the server headers look the same, why would it not be safe to decide those are duplicates of a single canonical url? It's not safe to merge www.domain1.com and www.domain2.com. it's not safe to merge subdomain.domain.com and www.domain.com. However, for the limited cases of www and no-www, https and http, if they look similar, I think it's harmful not to treat them as the same site. You can't expect every website owner to be aware of this issue. If it's a matter of not being able to be 100% sure, is there a single site that cares about google ranking that runs different sites on different combinations of www/no-www and https/http, but has similar content that would confuse a simple heuristic looking at page similarity? In what sort of circumstance could that happen other than with placeholder pages? GWT allows selecting a preference between www and no-www, but I don't see a preference between https and http. I think Google should add a notice that using GWT to select between www and no-www is deprecated and the recommended way to handle www, no-www, http, and https selection is to use 301 redirects or rel="canonical" tags.
rebelde 14y ago
And this is why many people like me refuse to allow https connections to their sites... Who wants to confuse Googlebot?
- saurik 14y ago
  Yeah... this is actually a surprising (at least to me; certainly irritating) liability of allowing users to access your website via https..
superchink 14y ago
I don't mean to derail the conversation, but I just noticed today that one of the sites I work with (aptcorner.com) has dropped off of the first page of results for the company name (it was previously at around position 4–5. Is it not enough that I've set the preferred domain in webmaster tools? Will setting a rel=canonical tag make a difference?

Codhisattva 14y ago

Wow! Kudos to Google putting someone out in the wild answering these kinds of questions. Is there a normal place to ask for web/index help or does HN serve that purpose?

Matt_Cutts 14y ago
http://productforums.google.com/forum/#!forum/webmasters http://productforums.google.com/forum/#!forum/webmasters Those folks would have spotted these issues pretty quickly.
pierrefar 14y ago
The best places are our forums ( http://productforums.google.com/forum/m/#!forum/webmasters http://productforums.google.com/forum/m/#!forum/webmasters ) and our regular webmaster office hours ( http://sites.google.com/site/webmasterhelpforum/en/office-hours http://sites.google.com/site/webmasterhelpforum/en/office-ho... ) where a webmaster support Googler (me and a few others) have a hangout on Google+ that anyone can join and ask about their site. We do them in many languages and time zones to cover the world as much as possible. Obviously we can't be everywhere and we can't answer every question, but we try as much as possible to help when we can.

Matt_Cutts 14y ago

Pierre, thanks for stopping by to confirm these issues. I often see sites be inconsistent between www and non-www, or between http and https. It looks like safeshepherd.com was doing both. More consistent redirects and adding rel=canonical should definitely help us figure out which url you prefer. Just to confirm what I said elsewhere, this site doesn't have any manual spam actions or anything like that. It's just a matter of Google trying to pick the correct canonical url when you have a lot of different (www, non-www, http, https) urls you're showing. If you make things more consistent, I think Google will stabilize on your preferred url pretty quickly.

jeebus 14y ago
Matt & Pierre, thanks for your thoughts and sorry that this ended up being a rookie mistake. I have a rel canonical good to go. Thanks again for your time.
- Matt_Cutts 14y ago
  No worries at all--glad it turned out to be easily fixable. And now I know what "nerfed" means. :)
  - kayge 14y ago
    FYI - Another fairly common usage of "nerfed" these days (especially in the gaming community) refers to something being toned-down. E.g. if many players are complaining about a character ability being too powerful, the developers may consider "nerfing" that character.
    - deleted 14y ago
      [deleted]
  - sixQuarks 14y ago
    This is why I love HackerNews. Guy asks for SEO help, frigg'n Matt Cutts answers!
  - 14y ago

harshreality 14y ago

Why doesn't google look at identical url paths on https, http, www, and no-www variants of the url, and if they look similar then use some default google policy to select which of them is canonical? For example, if http://mydomain.com/path http://mydomain.com/path and https://www.mydomain.com/path https://www.mydomain.com/path have 95% content correlation and repeated requests to http://mydomain.com/path http://mydomain.com/path have 95% content correlation, and the server headers look the same, why would it not be safe to decide those are duplicates of a single canonical url? It's not safe to merge www.domain1.com and www.domain2.com. it's not safe to merge subdomain.domain.com and www.domain.com. However, for the limited cases of www and no-www, https and http, if they look similar, I think it's harmful not to treat them as the same site. You can't expect every website owner to be aware of this issue. If it's a matter of not being able to be 100% sure, is there a single site that cares about google ranking that runs different sites on different combinations of www/no-www and https/http, but has similar content that would confuse a simple heuristic looking at page similarity? In what sort of circumstance could that happen other than with placeholder pages? GWT allows selecting a preference between www and no-www, but I don't see a preference between https and http. I think Google should add a notice that using GWT to select between www and no-www is deprecated and the recommended way to handle www, no-www, http, and https selection is to use 301 redirects or rel="canonical" tags.

rebelde 14y ago

And this is why many people like me refuse to allow https connections to their sites... Who wants to confuse Googlebot?

saurik 14y ago
Yeah... this is actually a surprising (at least to me; certainly irritating) liability of allowing users to access your website via https..

superchink 14y ago

I don't mean to derail the conversation, but I just noticed today that one of the sites I work with (aptcorner.com) has dropped off of the first page of results for the company name (it was previously at around position 4–5. Is it not enough that I've set the preferred domain in webmaster tools? Will setting a rel=canonical tag make a difference?