3 ms·
But robots.txt is just for exclusion directives, right? AFAICT it's not really used for assistance with finding resources.
by cpach 14y ago
But robots.txt is just for exclusion directives, right? AFAICT it's not really used for assistance with finding resources.
- tg3 14y agoThat's true, but you can also specify the Sitemap from the robots.txt, which is an inclusion standard.