Yeah, I assumed you meant machine resources. Machine resources tend to be the limiting factor for massively parallel things like crawling, so that's why I assum

Yeah, I assumed you meant machine resources. Machine resources tend to be the limiting factor for massively parallel things like crawling, so that's why I assumed that. My hypothesis is that there's a single crawler that was looking at my site, and it makes a best effort to crawl as much as it can. I have more pages on my site than can be crawled in a single day at 300ms/page, but at 30ms/page, it can be done in 3-4 hours. I don't know enough about their architecture to make any further guesses than to say that it probably just grabs as much as it can within the time it's focused on my site before something else on the queue becomes a higher priority.