3 ms·

what about meta entry(s) in the robots.txt, specifying if additional things like business.txt exist (and where)

• cpach 14y ago
  But robots.txt is just for exclusion directives, right? AFAICT it's not really used for assistance with finding resources.
  • tg3 14y ago
    That's true, but you can also specify the Sitemap from the robots.txt, which is an inclusion standard.