3 ms·
I applaud the effort, but the search results quality is really poor. It's basically an Altavista style laundry list of anything remotely relevant in no particul
by huhtenberg 12y ago
I applaud the effort, but the search results quality is really poor. It's basically an Altavista style laundry list of anything remotely relevant in no particular order.
That said, do you do your own crawling or do you source results from someone?
- deusu 12y agoI do the crawling myself. There are currently about 320 million pages in the search-index.
- frik 12y agoHow many GB of data are 320 million (HTML) pages? How long does it take to refresh the index (with a single crawler on a 1GB/s connection)?
- deusu 12y agoThe completed index takes up 337gb. The incoming data to the crawler... not so sure... something on the order of 10-20tb. I didn't really measure this. And I don't keep all the data. There is no "cache" function. On a 1gbit/s connection it takes about a week to crawl and generate the index.