4 ms·
I can't think of a single reason. Some of the URLs still work too.
by ben010783 12y ago
I can't think of a single reason. Some of the URLs still work too.
- thephyber 12y agoThe point of robots.txt "Disallow"s is to tell search engine crawlers not to crawl a URL (which could exist). It's not to tell search engines that URLs matching a pattern don't exist.