I tried this back in 2006 - mozdex (only a wikipedia article survives) - it’s not cheap. I was a fan of lucene which led to nutch and eventually hadoop. so lots

I tried this back in 2006 - mozdex (only a wikipedia article survives) - it’s not cheap. I was a fan of lucene which led to nutch and eventually hadoop. so lots of servers running hdfs doing map reduce jobs to compile and update indexes. No one in the end cared about open search… duck seems to do alright under the guise of security but most non major searches are just meta searches these days because of economies of scale being highly disadvantageous to any upcoming search - and people just don’t search like they used to either. i was spending 2500 a month just on indexers and had query traffic taken off my costs would have shot through the roof since you want query nodes to all be in memory cache and that was expensive back then. today i would have used some modern in memory distributed doc dbs instead of query masters with heavy block buffer caches. i learned a lot but lost my shirt :)