2 ms·This only applies to crawlers who are set to recurse into URLs.by kordlessagain 6y agoThis only applies to crawlers who are set to recurse into URLs.