2 ms·
Also depends on what you want to do when you crawl them. I guess if it's a million sites it is going to be median distribution of number of pages. But I don't k
by bryanrasmussen 8y ago
Also depends on what you want to do when you crawl them. I guess if it's a million sites it is going to be median distribution of number of pages. But I don't know that information is available anywhere. I think the http archive used to tell you the number of pages on sites, but now it just tells you page sizes.
https://httparchive.org/reports/state-of-the-web?start=latest&view=list https://httparchive.org/reports/state-of-the-web?start=lates...
of course page sizes might also be important dependent on how you crawl and what you do with the data.