5 ms·
The most interesting part of Quickwit for me has been that because it stores the index in object stores, you can theoretically abstract away a lot of the horizo
by jszymborski 3y ago
The most interesting part of Quickwit for me has been that because it stores the index in object stores, you can theoretically abstract away a lot of the horizontal scaling with things like MinIO or SeaweedFS that have S3-compatible APIs w/o having to go to the cloud.
Unfortunately, I think it's a large reason why it was made a design choice that indexed documents are immutable (IIRC), which doesn't work for my use case.
- francoismassot 3y agoSo yes, currently we only support infrequent deletes for GDPR reasons mainly. It's possible to add updates/deletes to Quickwit, but this is a lot of work, and for now, we have not prioritized this development. Do you mind sharing your use case?
- jszymborski 3y agoOh, just run-of-the-mill full-text search of web crawls. Some web pages update frequently, so I would need something that can handle that. I do understand that you seem to be targeting the log use-case of Elasticsearch moreso than the "Apache Solr" use-case. I would be in passing curious of what about the GDPR makes infrequent deletes a design choice. My understanding of GDPR is that the "right to be forgotten" aspects would if anything require deletions to be inexpensive.
- piperswe 3y agoI would assume that users requesting to be forgotten is a fairly infrequent occurrence compared to just about any other operation.
- jszymborski 3y agoTrue, but it can also require massive deletes/updates. I suppose their are ways to structure things to minimize this, though.
- fulmicoton 3y agoWeb crawl can be an ok use case actually. The idea then would be to "reindex" the world. It might seem ludicrous, but to give you an idea, indexing CommonCrawl takes about a day with 8vCPUs.