3 ms·
It's not the "same" crawler. Probably each thread or each cluster machine instance of the crawler hitting it independently.
by bot403 8mo ago
It's not the "same" crawler. Probably each thread or each cluster machine instance of the crawler hitting it independently.
- mrweasel 8mo agoI sincerely doubt that search engines run their crawlers on a single machine and they got it figured out.
- OliverGuy 8mo agoThat's still the same crawler system though. And it's lazy engineering to not build in something to track when you last requested a url. And it's quite a trivial feature at that.