4 ms·Of course not, the crawler doesn't even read robots.txt because it's "too difficult to implement"by PufPufPuf 4y agoOf course not, the crawler doesn't even read robots.txt because it's "too difficult to implement"