4 ms·Why wouldn't it consider the archived state of robots.txt?by pestaa 12y agoWhy wouldn't it consider the archived state of robots.txt?