4 ms·

Are you thinking of sitemap?

• JohnFen 1y ago
  No, sitemap serves a similar but different purpose. The design goal of robots.txt is to guide crawlers to the parts of the site that are worth crawling. That it's used as a very weak method of access control is a hack.