Elaborate? It is the best thing web has for crawling rules. Yea ofc crawling rules can be broken and violated but all serious internet search engines respect ro

Elaborate? It is the best thing web has for crawling rules. Yea ofc crawling rules can be broken and violated but all serious internet search engines respect robots.txt.