3 ms·
My guess is it's dutifully obeying it, not storing anything from the site and then exiting, without clearing the site from the crawl queue.
by RobotToaster 8mo ago
My guess is it's dutifully obeying it, not storing anything from the site and then exiting, without clearing the site from the crawl queue.