6 ms·

Check out Quickwit, it is briefly mentioned but I think mistakenly dismissed. They have been working on a similar concept for a few years and the results are ex

Check out Quickwit, it is briefly mentioned but I think mistakenly dismissed. They have been working on a similar concept for a few years and the results are excellent. It’s in no way mainly for logs as they claim, it is a general purpose cloud native search engine like the one they suggest, very well engineered. It is based on Tantivy, a Lucene alternative in Rust. I have extensive hands on experience with both and I highly recommend Tantivy, it’s just superior in every way now, such a pleasure to use, an ideal example of what Rust was designed for.

victor106 2y ago
Thanks for this info.
bomewish 2y ago
The big issue with tantivy I've found is that it only deals with immutable data. So it can't be used for anything you want to do CRUD on. This rules out a LOT of use cases. It's a real shame imo.
- oersted 2y ago
  It is indeed mostly designed for bulk indexing and static search. But it is not a strict limitation, frequent small inserts and updates are performant too. Deleting can be a bit awkward, you can only delete every document with a given term in a field, but if you use it on a unique id field it's just like a normal delete. Tantivy is a low-level library to build your own search engine (Quickwit), like Lucene, it's not a search engine in itself. Kind of like how DBs are built on top of Key-Value Stores. But you can definitely build a CRUD abstraction on top of it.
  - bomewish 2y ago
    Certainly one could build that on top of quickwit — which also doesn’t allow crud — but it’s not trivial. You need to insert a new record with changes, then delete the record you want to update. The docs instruct that the latter action is expensive and might take hours(!). Then one would need a separate process to ensure this all went down appropriately (say server crashes after insert of updated record but before delete succeeds). Meanwhile you’ve got almost identical records able to be searched. Just not very nice for anything involving CRUD. Please do advise if I’ve missed something here. I was really excited about using quickwit for a project but have gone with Meilisearch precisely for these reasons. Otherwise it would be quickwit all the way.
- pentlander 2y ago
  I’m pretty sure that Lucene is exactly the same, the segments it creates are immutable and Elastic is what handles a “mutable” view of the data. Which makes sense because Tantivy is like Lucene, not ES. https://lucene.apache.org/core/7_7_0/core/org/apache/lucene/index/package-summary.html#segments https://lucene.apache.org/core/7_7_0/core/org/apache/lucene/...
Semaphor 2y ago
> It’s in no way mainly for logs as they claim Where can I find more information on using it for user-facing search? The repository [0] starts with "Cloud-native search engine for observability (logs, traces, and soon metrics!)" and keeps talking about those. [0]: https://github.com/quickwit-oss/quickwit https://github.com/quickwit-oss/quickwit
- hovering_nox 2y ago
  I would say here: Features https://quickwit.io/docs/overview/introduction#key-features https://quickwit.io/docs/overview/introduction#key-features
- oersted 2y ago
  That just seems to be the market where search engines have the most obvious business case, Elasticsearch positioned themselves in the same way. But both are general-purpose full-text search engines perfectly capable of any serious search use-case. Their original breakout demo was on Common Crawl: https://common-crawl.quickwit.io/ https://common-crawl.quickwit.io/ But thanks for pointing it out, I hadn't looked at it in a few months, it looks like they significantly changed their pitch in the last year. I assume they got VC money and they need to deliver now.
  - AsianOtter 2y ago
    But the demo does not work. I tried "England is" and a few similar queries. It spends three seconds then shows that nothing is found.
    - oersted 2y ago
      I tried it once and it instantly showed no results, but then I tried it again and it returned results in <1s. Just try it with a bunch of queries, I think there's caching too so it's hard to gauge performance properly. The blog post about the demo is from 2021 and they haven't promoted it much since. I'm surprised that they even kept it online, according to the sidebar it was ~$810/month in AWS at the time.
erk__ 2y ago
I have been using Tantivy for Garfield comic search for a few years now, it has been really nice to use in all that time.
- jprd 2y ago
  I'm simultaneously intrigued and thinking this is a funny joke at the same time. If this isn't a joke, I would love an example.
  - erk__ 2y ago
    Luckily it is not a joke! Its been about I have had running in some capacity for some years by now through a couple of rewrites. At some point Discord added "auto-complete" for commands, this meant that I can do a live lookup and give users a list of comics where some piece of text is. My index is a bit out of date, but comics before September last year can be searched up. The search index lives fully in memory as it is not that big since it is only 17363 comics. This does mean that it is rebuilt every startup, but that does not take long compared to the month long uptime it usually has. Example of a search for "funny joke": https://imgur.com/a/J4sRhPJ https://imgur.com/a/J4sRhPJ Hosted bot: https://discord.com/application-directory/404364579645292564 https://discord.com/application-directory/404364579645292564 Source code: https://git.sr.ht/~erk/lasagna https://git.sr.ht/~erk/lasagna
ZeroCool2u 2y ago
Meili search is also a great option. https://www.meilisearch.com/docs/learn/resources/comparison_to_alternatives https://www.meilisearch.com/docs/learn/resources/comparison_...
- orthecreedence 2y ago
  Does Meili support object store backends?
  - derekperkins 2y ago
    No, and no clustering / HA yet I believe. It is awesome though with the right use case. We've had it in prod for a few years now
- notamy 2y ago
  Meilisearch is great when it works, but when it breaks it's a total nightmare. I've hit multiple bugs that destroyed my search index, I've hit multiple undocumented limits, ... that all required rebuilding my index from scratch and doing a lot of work to find what was actually going on to report it. It doesn't help that some of the errors it gives are incredibly non-specific and make it quite difficult to find what's actually breaking it. All of that said, I still use it because it has sucked less than the other search engines to run.
  - bomewish 2y ago
    Can you give any sense as to the conditions under which it just breaks like that? This sounds quite concerning!
lsowen 2y ago
Has anyone tried openobserve (https://github.com/openobserve/openobserve https://github.com/openobserve/openobserve)? How does it compare/contrast to Quickwit as an "Elasticsearch for logs" replacement?
grounder 2y ago
Quickwit indicates it is for immutable data and not to be used for mutable data. Is that the case in your experience?
fiedzia 2y ago
> Tantivy, it’s just superior in every way now It lacks tons of features ES and Solr have, most notably geo search, but what it does it does a lot faster.

victor106 2y ago

Thanks for this info.

bomewish 2y ago

The big issue with tantivy I've found is that it only deals with immutable data. So it can't be used for anything you want to do CRUD on. This rules out a LOT of use cases. It's a real shame imo.

oersted 2y ago
It is indeed mostly designed for bulk indexing and static search. But it is not a strict limitation, frequent small inserts and updates are performant too. Deleting can be a bit awkward, you can only delete every document with a given term in a field, but if you use it on a unique id field it's just like a normal delete. Tantivy is a low-level library to build your own search engine (Quickwit), like Lucene, it's not a search engine in itself. Kind of like how DBs are built on top of Key-Value Stores. But you can definitely build a CRUD abstraction on top of it.
- bomewish 2y ago
  Certainly one could build that on top of quickwit — which also doesn’t allow crud — but it’s not trivial. You need to insert a new record with changes, then delete the record you want to update. The docs instruct that the latter action is expensive and might take hours(!). Then one would need a separate process to ensure this all went down appropriately (say server crashes after insert of updated record but before delete succeeds). Meanwhile you’ve got almost identical records able to be searched. Just not very nice for anything involving CRUD. Please do advise if I’ve missed something here. I was really excited about using quickwit for a project but have gone with Meilisearch precisely for these reasons. Otherwise it would be quickwit all the way.
pentlander 2y ago
I’m pretty sure that Lucene is exactly the same, the segments it creates are immutable and Elastic is what handles a “mutable” view of the data. Which makes sense because Tantivy is like Lucene, not ES. https://lucene.apache.org/core/7_7_0/core/org/apache/lucene/index/package-summary.html#segments https://lucene.apache.org/core/7_7_0/core/org/apache/lucene/...

Semaphor 2y ago

> It’s in no way mainly for logs as they claim Where can I find more information on using it for user-facing search? The repository [0] starts with "Cloud-native search engine for observability (logs, traces, and soon metrics!)" and keeps talking about those. [0]: https://github.com/quickwit-oss/quickwit https://github.com/quickwit-oss/quickwit

hovering_nox 2y ago
I would say here: Features https://quickwit.io/docs/overview/introduction#key-features https://quickwit.io/docs/overview/introduction#key-features
oersted 2y ago
That just seems to be the market where search engines have the most obvious business case, Elasticsearch positioned themselves in the same way. But both are general-purpose full-text search engines perfectly capable of any serious search use-case. Their original breakout demo was on Common Crawl: https://common-crawl.quickwit.io/ https://common-crawl.quickwit.io/ But thanks for pointing it out, I hadn't looked at it in a few months, it looks like they significantly changed their pitch in the last year. I assume they got VC money and they need to deliver now.
- AsianOtter 2y ago
  But the demo does not work. I tried "England is" and a few similar queries. It spends three seconds then shows that nothing is found.
  - oersted 2y ago
    I tried it once and it instantly showed no results, but then I tried it again and it returned results in <1s. Just try it with a bunch of queries, I think there's caching too so it's hard to gauge performance properly. The blog post about the demo is from 2021 and they haven't promoted it much since. I'm surprised that they even kept it online, according to the sidebar it was ~$810/month in AWS at the time.

erk__ 2y ago

I have been using Tantivy for Garfield comic search for a few years now, it has been really nice to use in all that time.

jprd 2y ago
I'm simultaneously intrigued and thinking this is a funny joke at the same time. If this isn't a joke, I would love an example.
- erk__ 2y ago
  Luckily it is not a joke! Its been about I have had running in some capacity for some years by now through a couple of rewrites. At some point Discord added "auto-complete" for commands, this meant that I can do a live lookup and give users a list of comics where some piece of text is. My index is a bit out of date, but comics before September last year can be searched up. The search index lives fully in memory as it is not that big since it is only 17363 comics. This does mean that it is rebuilt every startup, but that does not take long compared to the month long uptime it usually has. Example of a search for "funny joke": https://imgur.com/a/J4sRhPJ https://imgur.com/a/J4sRhPJ Hosted bot: https://discord.com/application-directory/404364579645292564 https://discord.com/application-directory/404364579645292564 Source code: https://git.sr.ht/~erk/lasagna https://git.sr.ht/~erk/lasagna

ZeroCool2u 2y ago

Meili search is also a great option. https://www.meilisearch.com/docs/learn/resources/comparison_to_alternatives https://www.meilisearch.com/docs/learn/resources/comparison_...

orthecreedence 2y ago
Does Meili support object store backends?
- derekperkins 2y ago
  No, and no clustering / HA yet I believe. It is awesome though with the right use case. We've had it in prod for a few years now
notamy 2y ago
Meilisearch is great when it works, but when it breaks it's a total nightmare. I've hit multiple bugs that destroyed my search index, I've hit multiple undocumented limits, ... that all required rebuilding my index from scratch and doing a lot of work to find what was actually going on to report it. It doesn't help that some of the errors it gives are incredibly non-specific and make it quite difficult to find what's actually breaking it. All of that said, I still use it because it has sucked less than the other search engines to run.
- bomewish 2y ago
  Can you give any sense as to the conditions under which it just breaks like that? This sounds quite concerning!

lsowen 2y ago

Has anyone tried openobserve (https://github.com/openobserve/openobserve https://github.com/openobserve/openobserve)? How does it compare/contrast to Quickwit as an "Elasticsearch for logs" replacement?

grounder 2y ago

Quickwit indicates it is for immutable data and not to be used for mutable data. Is that the case in your experience?

fiedzia 2y ago

> Tantivy, it’s just superior in every way now It lacks tons of features ES and Solr have, most notably geo search, but what it does it does a lot faster.