4 ms·
I'm starting to wonder if people are interpreting "resources" to mean "CPU resources," when I really mean "resources" as in time, money, etc. What's your hypot
by kadavy 16y ago
I'm starting to wonder if people are interpreting "resources" to mean "CPU resources," when I really mean "resources" as in time, money, etc.
What's your hypothesis for why the crawler started indexing more pages per day when your response time improved? That seems like evidence of my theory, but I'm a designer by training so maybe there's a more technical explanation I'm missing.
- ericd 16y agoYeah, I assumed you meant machine resources. Machine resources tend to be the limiting factor for massively parallel things like crawling, so that's why I assumed that. My hypothesis is that there's a single crawler that was looking at my site, and it makes a best effort to crawl as much as it can. I have more pages on my site than can be crawled in a single day at 300ms/page, but at 30ms/page, it can be done in 3-4 hours. I don't know enough about their architecture to make any further guesses than to say that it probably just grabs as much as it can within the time it's focused on my site before something else on the queue becomes a higher priority.