3 ms·Regardless. You should still respect robots.txt..by zendist 6mo agoRegardless. You should still respect robots.txt..