3 ms·If they don't comply with robots.txt, why would they comply with anything else?by jsheard 2y agoIf they don't comply with robots.txt, why would they comply with anything else?