3 ms·Exactly. You disobey robots file => we'll make your crawl gain a net negative.by xyzal 7mo agoExactly. You disobey robots file => we'll make your crawl gain a net negative.