3 ms·
I think this post has a grain of truth behind it, though I don't agree with the reasoning about resource utilization. I noticed that when one of my sites went
by ericd 17y ago
I think this post has a grain of truth behind it, though I don't agree with the reasoning about resource utilization.
I noticed that when one of my sites went from 300ms response times to 30ms, the crawler started indexing more pages per day, and would index deeper, which meant more of my pages in their index. The result was a healthy boost in organic search traffic due to more long tail search matches.
- kadavy 17y agoI'm starting to wonder if people are interpreting "resources" to mean "CPU resources," when I really mean "resources" as in time, money, etc. What's your hypothesis for why the crawler started indexing more pages per day when your response time improved? That seems like evidence of my theory, but I'm a designer by training so maybe there's a more technical explanation I'm missing.
- ericd 17y agoYeah, I assumed you meant machine resources. Machine resources tend to be the limiting factor for massively parallel things like crawling, so that's why I assumed that. My hypothesis is that there's a single crawler that was looking at my site, and it makes a best effort to crawl as much as it can. I have more pages on my site than can be crawled in a single day at 300ms/page, but at 30ms/page, it can be done in 3-4 hours. I don't know enough about their architecture to make any further guesses than to say that it probably just grabs as much as it can within the time it's focused on my site before something else on the queue becomes a higher priority.