5 ms·

robots.txt is a joke tho.

• mrkramer 5y ago
  Elaborate? It is the best thing web has for crawling rules. Yea ofc crawling rules can be broken and violated but all serious internet search engines respect robots.txt.