2 ms·
It's intended as a way to help robots crawl a site. If you have a robot that doesn't need help it should ignore the file. Because so many people misunderstand
by EdHominem 10y ago
It's intended as a way to help robots crawl a site. If you have a robot that doesn't need help it should ignore the file.
Because so many people misunderstand it, robots.txt is often a list of what you should crawl to find the good stuff.