4 ms·
OP, I'm not an expert but I think you need to introduce more entropy to prevent opportunistic crawling of every possible URL.
by mmahemoff 12y ago
OP, I'm not an expert but I think you need to introduce more entropy to prevent opportunistic crawling of every possible URL.