9 ms·Crawlers were suppose to have this down in 2012 (at least google),now they recommend a sitemap page.by wolco 7y agoCrawlers were suppose to have this down in 2012 (at least google),now they recommend a sitemap page.