9 ms·

I was curious what it was complaining about, since https://henryfjordan.com https://henryfjordan.com looks great to me. I tried to run it through Google's "Mob

by jefftk 7y ago

I was curious what it was complaining about, since https://henryfjordan.com https://henryfjordan.com looks great to me. I tried to run it through Google's "Mobile Friendly Test" but fetching failed [1] because your robots.txt has: User-agent: * Disallow: / This would explain why you've gotten zero clicks from Google (or I would guess anyone else's) search results! On the other hand, it's surprising that you would get a notification if you had crawling disabled. Did you set this robots.txt up recently? [1] https://search.google.com/test/mobile-friendly?id=97_WUiIxx-inLfwyqmbHuQ https://search.google.com/test/mobile-friendly?id=97_WUiIxx-... (Disclosure: I work at Google, commenting only for myself)

SahAssar 7y ago
Google seems to see robots.txt as "more what you call guidelines, than actual rules". Sites that block googlebot or all bots with robots.txt still turn up in google searches, just without a description, and are obviously still indexed.
- dorgo 7y ago
  >"more what you call guidelines, than actual rules" they can index without scraping. It is enough that other websites have links to you site. So the google bot follows the rules in robots.txt to the letter. "no-index" is the way to stay away from google.
  - SahAssar 7y ago
    They can't read my no-index if they obey my robots.txt. Do they break the robots.txt to be able to read my no-index or do they assume my "Disallow: /" means I'm fine with them indexing/linking? Without the noindex part of robots.txt (which google decided to ignore not so long ago) this is not solvable.
- jefftk 7y ago
  robots.txt is a tool to control crawling, not to specify how you would like your site to be displayed (or not) in search results. If you don't want search engines to include your site, set: <meta name="robots" content="noindex"> while to block just Google do: <meta name="googlebot" content="noindex"> See https://support.google.com/webmasters/answer/93710 https://support.google.com/webmasters/answer/93710 If Googlebot is not respecting robots.txt, and is crawling something it's been instructed not to crawl, let me know and I can file a bug? (Disclosure: I work for Google but not on Search, speaking only for myself)
  - areyousure 7y ago
    But that requires that Googlebot be allowed to crawl the page in robots.txt in the first place. How do you tell Googlebot to not crawl your site and to not index it either? Previously, one could use the undocumented "Noindex" directive in robots.txt, but this will be disabled soon: https://webmasters.googleblog.com/2019/07/a-note-on-unsupported-rules-in-robotstxt.html https://webmasters.googleblog.com/2019/07/a-note-on-unsuppor...
    - jefftk 7y ago
      The blog post you link has a bunch of alternatives, but I agree they're not great. If there are a lot of webmasters who want to be able to noindex through robots.txt then making the case for adding noindex to the standard would be a good next step. (Still speaking only for myself)
      - ademarre 7y ago
        Googlebot actually used to support a noindex rule in robots.txt, but they are removing it. https://webmasters.googleblog.com/2019/07/a-note-on-unsupported-rules-in-robotstxt.html https://webmasters.googleblog.com/2019/07/a-note-on-unsuppor...
        jefftk 7y ago
        Yes, that was linked above. It looks like this is part of reducing support to what's in the spec?
        ademarre 7y ago
        Oops, yep. I didn't see that context.
    - tylerl 7y ago
      The bot doesn't need to crawl your site for it to be indexed; it crawls other sites that link to yours. You can specify your index preferences in Webmaster Tools. Don't know if there's a domain-wide off switch in there, but there probably is.
henryfjordan 7y ago
Oh, I just added that yesterday as a response to the email. Before that I was actually running Google Analytics but since I get basically 0 clicks it wasn't really useful. I have a feeling the PDF viewer triggered it, cause on Mobile it defaults to showing the whole page which results in tiny text but that's easily fixed by the user so I prefer to leave it like that.

by jefftk 7y ago

SahAssar 7y ago

Google seems to see robots.txt as "more what you call guidelines, than actual rules". Sites that block googlebot or all bots with robots.txt still turn up in google searches, just without a description, and are obviously still indexed.

dorgo 7y ago
>"more what you call guidelines, than actual rules" they can index without scraping. It is enough that other websites have links to you site. So the google bot follows the rules in robots.txt to the letter. "no-index" is the way to stay away from google.
- SahAssar 7y ago
  They can't read my no-index if they obey my robots.txt. Do they break the robots.txt to be able to read my no-index or do they assume my "Disallow: /" means I'm fine with them indexing/linking? Without the noindex part of robots.txt (which google decided to ignore not so long ago) this is not solvable.
jefftk 7y ago
robots.txt is a tool to control crawling, not to specify how you would like your site to be displayed (or not) in search results. If you don't want search engines to include your site, set: <meta name="robots" content="noindex"> while to block just Google do: <meta name="googlebot" content="noindex"> See https://support.google.com/webmasters/answer/93710 https://support.google.com/webmasters/answer/93710 If Googlebot is not respecting robots.txt, and is crawling something it's been instructed not to crawl, let me know and I can file a bug? (Disclosure: I work for Google but not on Search, speaking only for myself)
- areyousure 7y ago
  But that requires that Googlebot be allowed to crawl the page in robots.txt in the first place. How do you tell Googlebot to not crawl your site and to not index it either? Previously, one could use the undocumented "Noindex" directive in robots.txt, but this will be disabled soon: https://webmasters.googleblog.com/2019/07/a-note-on-unsupported-rules-in-robotstxt.html https://webmasters.googleblog.com/2019/07/a-note-on-unsuppor...
  - jefftk 7y ago
    The blog post you link has a bunch of alternatives, but I agree they're not great. If there are a lot of webmasters who want to be able to noindex through robots.txt then making the case for adding noindex to the standard would be a good next step. (Still speaking only for myself)
    - ademarre 7y ago
      Googlebot actually used to support a noindex rule in robots.txt, but they are removing it. https://webmasters.googleblog.com/2019/07/a-note-on-unsupported-rules-in-robotstxt.html https://webmasters.googleblog.com/2019/07/a-note-on-unsuppor...
      - jefftk 7y ago
        Yes, that was linked above. It looks like this is part of reducing support to what's in the spec?
        ademarre 7y ago
        Oops, yep. I didn't see that context.
  - tylerl 7y ago
    The bot doesn't need to crawl your site for it to be indexed; it crawls other sites that link to yours. You can specify your index preferences in Webmaster Tools. Don't know if there's a domain-wide off switch in there, but there probably is.

henryfjordan 7y ago

Oh, I just added that yesterday as a response to the email. Before that I was actually running Google Analytics but since I get basically 0 clicks it wasn't really useful. I have a feeling the PDF viewer triggered it, cause on Mobile it defaults to showing the whole page which results in tiny text but that's easily fixed by the user so I prefer to leave it like that.