4 ms·
I love this idea. It has worked extremely well for Observability data, and more generally I’ve realized this really works for any append-only workload. I’d love
by brancz 5y ago
I love this idea. It has worked extremely well for Observability data, and more generally I’ve realized this really works for any append-only workload. I’d love to understand more what is going on under the hood. I imagine cache oblivious data structures but I’m having a hard time finding what the exact methods are that are being used to achieve this (maybe I just need to learn more about tantivity).
Anyhow, it’s an early project so I assume details will get better over time. Good luck, I can’t wait to see where this goes!
- francoismassot 5y agothank you! We solved 3 challenges as described in the blog post. The first one is really gathering metadata of different index files to know in advance where to read exactly the index data that will be needed when receiving a query. We call this the "hotcache". The hotcache is specific to quickwit. Then we also modified the index structure so that we only need to make only 3 subsequent queries on the storage to fetch the relevant data for return the matching documents. The index structure is coded in tantivy. Finally, with parallelisation of queries, we get the right throughput from s3. We definetetly need to write a blog post on that and explain what we did in some meetups and confs. PS: it's tantivy not tantivity :)