4 ms·
Ask HN: how much does it cost to crawl 1M sites?
What is the cheapest api or other solution ?
How much more does it cost to crawl sites with javascript ?
thx
- bufferoverflow 8y agoDepends on which sites. Some of my websites have 200K pages. Some have just three.
- bryanrasmussen 8y agoAlso depends on what you want to do when you crawl them. I guess if it's a million sites it is going to be median distribution of number of pages. But I don't know that information is available anywhere. I think the http archive used to tell you the number of pages on sites, but now it just tells you page sizes. https://httparchive.org/reports/state-of-the-web?start=latest&view=list https://httparchive.org/reports/state-of-the-web?start=lates... of course page sizes might also be important dependent on how you crawl and what you do with the data.
- visox 8y agoI think i made an unclear/wrong question. I wish to crawl about 1M pages (rather then sites) so 1M http get requests. I was hopping there would be some solution for 100$ lets say
- segmondy 8y ago$0. Crawl from your computer at home.