My opinion is to view it as an optional thing to help other crawlers( and bored humans) index your content. You can't really use it to block crawlers since bad

My opinion is to view it as an optional thing to help other crawlers( and bored humans) index your content. You can't really use it to block crawlers since bad crawlers will just ignore it. For example, if I wanted to download every single one of Microsoft knowledge base article, I could go to https://support.microsoft.com/robots.txt https://support.microsoft.com/robots.txt and download every url in the sitemap. Compare that with other sites that don't provide a sitemap so I would have to hit every single page and try to find out all the links manually by downloading, parsing for links, etc. Less work for me and less load for the site. While I"m not sure if it has a direct impact on seo, Making your site more accessible is only a good thing imho.