4 ms·

How does someone economically store tens if thousands of terabytes of data needed for the indexes of a large scale search engine? And have the large server ins

• supernovae 5y ago
  I tried this back in 2006 - mozdex (only a wikipedia article survives) - it’s not cheap. I was a fan of lucene which led to nutch and eventually hadoop. so lots of servers running hdfs doing map reduce jobs to compile and update indexes. No one in the end cared about open search… duck seems to do alright under the guise of security but most non major searches are just meta searches these days because of economies of scale being highly disadvantageous to any upcoming search - and people just don’t search like they used to either. i was spending 2500 a month just on indexers and had query traffic taken off my costs would have shot through the roof since you want query nodes to all be in memory cache and that was expensive back then. today i would have used some modern in memory distributed doc dbs instead of query masters with heavy block buffer caches. i learned a lot but lost my shirt :)
  • aantix 5y ago
    I think I would just start with indexing the latest Common Crawl WET files along with a server and a couple of Exos X20 20TB drives. Wondering out loud if there's a crawl that will produce real-time updates via common news outlets..
    • supernovae 5y ago
      Crawling is the easy part. The 20tb drives would be too slow for query though. Query really needs everything in memory to be fast enough.
• marginalia_nu 5y ago
  Why would you need that much data? The average website has maybe 10kB worth of textual information without compression. To get tens of thousands of terabytes of data, you'd need to index of the order 10^12 websites. That seems a bit much.