3 ms·
Depends on which sites. Some of my websites have 200K pages. Some have just three.
by bufferoverflow 8y ago
Depends on which sites. Some of my websites have 200K pages. Some have just three.
- bryanrasmussen 8y agoAlso depends on what you want to do when you crawl them. I guess if it's a million sites it is going to be median distribution of number of pages. But I don't know that information is available anywhere. I think the http archive used to tell you the number of pages on sites, but now it just tells you page sizes. https://httparchive.org/reports/state-of-the-web?start=latest&view=list https://httparchive.org/reports/state-of-the-web?start=lates... of course page sizes might also be important dependent on how you crawl and what you do with the data.