5 ms·

Are sitemaps still a thing that really matters?

by jkaljundi 9y ago

miguelrochefort 9y ago
Don't you wanna get crawled?
- irrational 9y ago
  Actually I do work on a website that we do not want crawled. It would be handy to know which things to leave out to make it less likely your site will be crawled or show up high in a google result.
  - mileycyrusXOXO 9y ago
    Just define a robot.txt file instructing the crawler to not index your website. See: http://www.robotstxt.org/ http://www.robotstxt.org/
    - seanwilson 9y ago
      This is a common misconception: https://support.google.com/webmasters/answer/7424835?hl=en#h17 https://support.google.com/webmasters/answer/7424835?hl=en#h... > robots.txt Disallow does not guarantee that a page will not appear in results: Google may still decide, based on external information such as incoming links, that it is relevant. If you wish to explicitly block a page from being indexed, you should instead use the noindex robots meta tag or X-Robots-Tag HTTP header. If it's something like a staging site, you'd be better putting it behind a login screen to prevent mistakes anyway.
- jkaljundi 9y ago
  Have not found the crawl rates changed for sites where everything is well linked together. Then again, these have been just small sites with a few million pages.
  - franze 9y ago
    measurability crawling != indexing with a segmented sitemap (i.e.: one per pagetype, per namespace, ..) you get important communicated to indexed ratios you otherwise do not get. if crawling and indexing works, it works but if it doesn't you should know it (see it in google search console) and debug it (narrow it down via segmented sitemaps).
    - andrei_says_ 9y ago
      Curious if you have examples of the types of sitemaps you mentioned: segmentation, namespacing, one per pagetype etc. and the reasoning behind the variations.
      - franze 9y ago
        i.e.: tourradar.com/robots.txt
hayksaakian 9y ago
If you're coding it from scratch, don't bother. If you're using wordpress or another pre-baked system, then it's worth having a sitemap plugin (example: Yoast SEO) or the built-in sitemap (like for Shopify). Submit it to Google Search Console, Bing, and you're done (essentially).

Are sitemaps still a thing that really matters?

miguelrochefort 9y ago

Don't you wanna get crawled?

irrational 9y ago
Actually I do work on a website that we do not want crawled. It would be handy to know which things to leave out to make it less likely your site will be crawled or show up high in a google result.
- mileycyrusXOXO 9y ago
  Just define a robot.txt file instructing the crawler to not index your website. See: http://www.robotstxt.org/ http://www.robotstxt.org/
  - seanwilson 9y ago
    This is a common misconception: https://support.google.com/webmasters/answer/7424835?hl=en#h17 https://support.google.com/webmasters/answer/7424835?hl=en#h... > robots.txt Disallow does not guarantee that a page will not appear in results: Google may still decide, based on external information such as incoming links, that it is relevant. If you wish to explicitly block a page from being indexed, you should instead use the noindex robots meta tag or X-Robots-Tag HTTP header. If it's something like a staging site, you'd be better putting it behind a login screen to prevent mistakes anyway.
jkaljundi 9y ago
Have not found the crawl rates changed for sites where everything is well linked together. Then again, these have been just small sites with a few million pages.
- franze 9y ago
  measurability crawling != indexing with a segmented sitemap (i.e.: one per pagetype, per namespace, ..) you get important communicated to indexed ratios you otherwise do not get. if crawling and indexing works, it works but if it doesn't you should know it (see it in google search console) and debug it (narrow it down via segmented sitemaps).
  - andrei_says_ 9y ago
    Curious if you have examples of the types of sitemaps you mentioned: segmentation, namespacing, one per pagetype etc. and the reasoning behind the variations.
    - franze 9y ago
      i.e.: tourradar.com/robots.txt

hayksaakian 9y ago

If you're coding it from scratch, don't bother. If you're using wordpress or another pre-baked system, then it's worth having a sitemap plugin (example: Yoast SEO) or the built-in sitemap (like for Shopify). Submit it to Google Search Console, Bing, and you're done (essentially).