3 ms·
If two customers crawl the same website and it uses crawl-delay, how does it handle that? Are they independent, or does each one run half as fast?
by skybrian 8mo ago
If two customers crawl the same website and it uses crawl-delay, how does it handle that? Are they independent, or does each one run half as fast?
- PeterStuer 8mo agoYou put a governor on the domain, and you return from the cache instead.