2 ms·
When spiders use more bandwidth than customers, your website must not be very popular. It implies that each page is viewed only a handful of times / month on av
by tlb 15y ago
When spiders use more bandwidth than customers, your website must not be very popular. It implies that each page is viewed only a handful of times / month on average.
- tlrobinson 15y agoIt's also possible they have a huge amount of content that is sparsely accessed by a large number of users. But in general I agree. Edit: SmugMug seems to fall into this category: http://don.blogs.smugmug.com/2010/07/15/great-idea-google-should-open-their-index/ http://don.blogs.smugmug.com/2010/07/15/great-idea-google-sh... Also interesting: And if you think about it, the robots are much harder to optimize for – they’re crawling the long tail, which totally annihilates your caching layers. Humans are much easier to predict and optimize for.