4 ms·
Show HN: Highly cost-efficient search engine in Rust
- brancz 5y agoI love this idea. It has worked extremely well for Observability data, and more generally I’ve realized this really works for any append-only workload. I’d love to understand more what is going on under the hood. I imagine cache oblivious data structures but I’m having a hard time finding what the exact methods are that are being used to achieve this (maybe I just need to learn more about tantivity). Anyhow, it’s an early project so I assume details will get better over time. Good luck, I can’t wait to see where this goes!
- francoismassot 5y agothank you! We solved 3 challenges as described in the blog post. The first one is really gathering metadata of different index files to know in advance where to read exactly the index data that will be needed when receiving a query. We call this the "hotcache". The hotcache is specific to quickwit. Then we also modified the index structure so that we only need to make only 3 subsequent queries on the storage to fetch the relevant data for return the matching documents. The index structure is coded in tantivy. Finally, with parallelisation of queries, we get the right throughput from s3. We definetetly need to write a blog post on that and explain what we did in some meetups and confs. PS: it's tantivy not tantivity :)
- mikevm 5y agoThis looks pretty cool, but what is the business model here?
- francoismassot 5y agoWe started with the idea of a double license model: AGPL or commercial. A saas model is a possibility but we are too far from it today.
- shamig_g 5y agogreat work.
- francoismassot 5y agoHi HN! After months of quiet work, we are releasing today our first version of Quickwit. We wrote a blog post about how we improve cost efficiency of traditional search engines: https://quickwit.io/blog/quickwit-first-release https://quickwit.io/blog/quickwit-first-release