I've scaled a cluster that was in the 100s of millions of results range, the experience was not great and tuning for our use case which was decidedly not a typi

I've scaled a cluster that was in the 100s of millions of results range, the experience was not great and tuning for our use case which was decidedly not a typical enterprise search problem and that made it a complete pain. So that's great that it works for that particular case, and we ultimately made it work ourselves much like you're suggesting, we used vector search with something like FAISS as a pre filtering step and then a final search through a much reduced set of ids in ES but it's pretty clear a new player could come in here and make a much better experience. Basically ES is, if not unsuitable, a big pain for large non enterprise search such as web search, where things like vector search are one major signal and provide a better search experience. And that's the exact problem there aren't off the shelf open source solutions if you're not doing a fairly standard ecom or internal business search type problem like log aggregation or internal documents. I'm also suggesting that use cases that aren't enterprise type search problems are more common than you'd think these days. Edit: Additionally the thing here is you have classical boolean search systems like ES and vector search solutions like Milvus, but no one's gotten around to making something that does both well, from what I can see a lot of the players in the space are trying to go in that direction but it's a slow painful crawl that results in this type of situation where we had to do a lot of custom gluing of these systems together and keeping that parity that was super annoying and expensive, and time consuming, but not necessarily performance inhibiting.