3 ms·Special-case an exclusion for any site that contains robots.txt?by pkamb 16y agoSpecial-case an exclusion for any site that contains robots.txt?