3 ms·
I do the crawling myself. There are currently about 320 million pages in the search-index.
by deusu 12y ago
I do the crawling myself. There are currently about 320 million pages in the search-index.
- frik 12y agoHow many GB of data are 320 million (HTML) pages? How long does it take to refresh the index (with a single crawler on a 1GB/s connection)?
- deusu 12y agoThe completed index takes up 337gb. The incoming data to the crawler... not so sure... something on the order of 10-20tb. I didn't really measure this. And I don't keep all the data. There is no "cache" function. On a 1gbit/s connection it takes about a week to crawl and generate the index.