4 ms·
I've crawled over a billion pages over a stretch of 3 years or so. Crawling is the easy task and just crawling a billion pages wouldn't cost more than a few tho
by AznHisoka 10y ago
I've crawled over a billion pages over a stretch of 3 years or so. Crawling is the easy task and just crawling a billion pages wouldn't cost more than a few thousand a month. Add a couple more thousand for storing these pages in a search index and database.
- outpan 10y agoWould you be able to share what your stack was? and the resources it took? Thanks a lot.
- usgroup 10y agoJust in case you don't know common-crawl makes available a huge crawl dataset
- outpan 10y agoCommon Crawl is great! however, some use cases require larger crawls with a higher frequency.
- AznHisoka 10y agoRuby and Sidekiq as the messaging queue Postgres to store the data Elasticsearch as a search index. My ES cluster has around 10 nodes, 64 GB RAM, quad-core. Postgres database cluster is 4 nodes, 1 TB, 64 GB RAM, quad-core. 800 crawler threads distributed across 10 dedicated servers.
- theviajerock 10y agoDo you have a company that does this ? Can you advice me about this? I like the crawling thing, I would like to know how to monetize this. Thanks!