12 ms·

Sonic: Fast, lightweight and schemaless search back end in Rust

238 points by louis-paul 8y ago

FridgeSeal 8y ago
Talk about perfect timing! I was looking for something just like this for a project in my team. We had been using this setup where a huge chunk of the data was being stored in triplicate: some of it in ES, some more of it in another database and finally the whole dataset in our data warehouse. Hopefully I can use this to only provide the index + full text capability and just use the warehouse itself as the main db because the query performance is similar enough and the warehouse is criminally underused for what we pay for it.
- dannycastonguay 8y ago
  Curious to know if you considered https://www.algolia.com/ https://www.algolia.com/ as well?
  - mellab 8y ago
    Algolia is a beautiful product but it’s expensive. At just 1 mil items you’re already paying 500 a month
- winrid 8y ago
  What's preventing you from using ES for everything? Slow writes?
  - FridgeSeal 8y ago
    Write speed is fine, it’s more the fact that the dataset is reasonably large, and to run an instance with enough capacity and nodes (even with spill to disk), is silly expensive.
    - mverwijs 8y ago
      Being in a simular situation what do you consider "reasonably large"?
      - FridgeSeal 8y ago
        250-300GB. Not large by absolute standards sure, but large enough to cause issues. I’m sure there’s some kind of solution that involves re-architecting the ES cluster and indices and re-architecting the data flows and stuff. But if our options are go through all that, or seriously slim down our architecture and costs by just running Sonic + our data warehouse, I’m definitely going to give it a go. After all, worst comes to worst we can go down the re-architecting ES route if Sonic doesn’t work out. ¯\_(ツ)_/¯
        Xylakant 8y ago
        I’d be curious what your expectations and constraints are, but from my experience of running clusters in the double digit TB-Size my ballpark figure for that amount of data would be 2 medium size data nodes and a small tiebreaker. Alternatively, if you can live with the reduced resilience and availability, even a single node might just do. Depends on the expectated churn though, ES really does not like document updates.
        FridgeSeal 8y ago
        Is that double digit TB on ElasticSearch?
        Xylakant 8y ago
        Yes, certainly.
        cinbun8 8y ago
        That does not sound like a good idea. You can't even maintain a quorum of 2 replicas with n=3 on a cluster like that. Losing one data node would be disastrous.
StavrosK 8y ago
This looks pretty cool! I'd like to try it, but it looks like I'll have to wait for the Python client library first.
- deadwisdom 8y ago
  Wanna build it with me?
  - arcticbull 8y ago
    The Rust Python bindings are actually pretty good, I hacked together a project to let you deploy Rust micro services in Lambda via Python module bindings a while back.
  - StavrosK 8y ago
    Normally I would, but I don't have an immediate use for this, which kills my motivation
- marmaduke 8y ago
  The protocol description makes it look fairly trivial to script, https://github.com/valeriansaliou/sonic/blob/master/PROTOCOL.md https://github.com/valeriansaliou/sonic/blob/master/PROTOCOL...
- 21stio 8y ago
  a grpc endpoint would be nice
- MosheZada 8y ago
  I just wrote a basic one https://github.com/moshe/asonic https://github.com/moshe/asonic
  - StavrosK 8y ago
    That looks great, thank you! I'll try it out and send you any feedback I have. Also, you might want to link to Sonic in the README, for people unfamiliar with it.
mrec 8y ago
Huh. When this was initially posted it had a weird and commercially restrictive license, but it looks like that's been reverted, possibly after (polite) discussion on /r/rust. It's MPL 2 now. https://github.com/valeriansaliou/sonic/issues/52#issuecomment-475911660 https://github.com/valeriansaliou/sonic/issues/52#issuecomme...
sidcool 8y ago
I would like to understand the code for the project. What approach would help?
- d33 8y ago
  Personally, I would advise finding an itch to scratch - something you'd like to see improved. Then try to understand the code from this perspective - where would you put the functionality, which pieces would it be connected to? Try to follow the lead, make notes as you read the code. You'll eventually get a feel of the infrastructure, going from the deal to the big picture - this is my default way of navigating projects. If at any moment you feel lost, look through related issues and merge requests, as well as Git history. Perhaps you'll see how things get changed in the project, patterns intrinsic to it. Also, keep in mind that you can always try to contact the community/author or invite someone to try figuring out your goal with you - once you collaborate, you'll get a solution tailored to the way you think. It does engage other people, but it also makes coding social and (at least to me) more satisfying. Let me know what you think of this approach!
mellab 8y ago
Wow. I’ve spent the last three weeks building a custom search solution in Kotlin - in my case I’m using tokenizers from Lucene and using a radix trie as an index. I actually looked at using Bleve (another rust search lib) initially but it didn’t have the right language support Glancing over this it looks like a nearly perfect fit for my use case I just wish I had seen this a couple of weeks earlier!
- O_H_E 8y ago
  And that's why I believe that the search/discovery problem is not solved yet by google
  - StavrosK 8y ago
    Far from it, search sucks nowadays. Just think about how hard it is to find stuff from Twitter, reddit etc, even though that's where most of the content is. And if you don't know what you want exactly, you can't find anything.
  - manigandham 8y ago
    Google is a public service to search the internet. This product and thread is more about adding search functionality to other applications and private data.
- networkimprov 8y ago
  Bleve, from Couchbase, is in Go :-) https://github.com/blevesearch/bleve https://github.com/blevesearch/bleve
marmaduke 8y ago
This looks like a breath of fresh air. Elasticsearch won’t even start if it can’t preallocate 2 GB
- Xylakant 8y ago
  That’s just false. The default config might set the JVM HEAP to 2GB (though I’m fairly certain it’s 1GB) but ES will start up with half a GB of heap with no issue.
  - marmaduke 8y ago
    It was the default behavior on my install, documented recommendation is half the system memory and the logs don’t provide useful info when the heap allocation fails. Perhaps the upstream default is 1 GB but this was not the default, and there is much confusion on setting these correctly https://stackoverflow.com/a/40333263 https://stackoverflow.com/a/40333263 Not everyone has time to dig into it; of course if you did that’s good for you
    - Xylakant 8y ago
      What your system install does and whether the default is good for what you intend to do is a bit different from what the system allows. If you want to use a piece of software, you’re expected to somewhat understand what it does. At least reading the “setup” chapter of the documentation which explains which ENV var to set should be recommended. (I specifically link to the 1.7 docs to show that it’s been in there for quite some time) https://www.elastic.co/guide/en/elasticsearch/reference/1.7/setup-configuration.html https://www.elastic.co/guide/en/elasticsearch/reference/1.7/... The recommendation is btw: at most half of the systems memory, no more than 31g (due to how the jvm compresses heap pointers), and keep at least as much memory available for disk buffers/caches as you allocate to the heap.
javitury 8y ago
Performance figures are awesome. Also language support is great. > Sonic only keeps the N most recently pushed results for a given word This index discards old entries. This is fine for messages, in which aging items lose relevance. Yet the developer uses it for a help desk, which I think should give equal importance to all items. In this area I would say the main comperitor is Groonga. It can be integrated into PostgreSQL with the PGroonga extension, and it indexes all of the data. However it consumes way more ram.
manigandham 8y ago
Always nice to see more alternatives to Elasticsearch. That project could be so much better with some proper planning and focus. There's also Toshi: https://github.com/toshi-search/Toshi https://github.com/toshi-search/Toshi which is built on top of Tantivy: https://github.com/tantivy-search/tantivy https://github.com/tantivy-search/tantivy And for C++, there's Xapiland: https://github.com/Kronuz/Xapiand https://github.com/Kronuz/Xapiand And for Go, there's Blast: https://github.com/mosuka/blast https://github.com/mosuka/blast built on Bleve: https://github.com/blevesearch/bleve https://github.com/blevesearch/bleve

238 points by louis-paul 8y ago

Talk about perfect timing! I was looking for something just like this for a project in my team. We had been using this setup where a huge chunk of the data was being stored in triplicate: some of it in ES, some more of it in another database and finally the whole dataset in our data warehouse. Hopefully I can use this to only provide the index + full text capability and just use the warehouse itself as the main db because the query performance is similar enough and the warehouse is criminally underused for what we pay for it.

dannycastonguay 8y ago
Curious to know if you considered https://www.algolia.com/ https://www.algolia.com/ as well?
- mellab 8y ago
  Algolia is a beautiful product but it’s expensive. At just 1 mil items you’re already paying 500 a month
winrid 8y ago
What's preventing you from using ES for everything? Slow writes?
- FridgeSeal 8y ago
  Write speed is fine, it’s more the fact that the dataset is reasonably large, and to run an instance with enough capacity and nodes (even with spill to disk), is silly expensive.
  - mverwijs 8y ago
    Being in a simular situation what do you consider "reasonably large"?
    - FridgeSeal 8y ago
      250-300GB. Not large by absolute standards sure, but large enough to cause issues. I’m sure there’s some kind of solution that involves re-architecting the ES cluster and indices and re-architecting the data flows and stuff. But if our options are go through all that, or seriously slim down our architecture and costs by just running Sonic + our data warehouse, I’m definitely going to give it a go. After all, worst comes to worst we can go down the re-architecting ES route if Sonic doesn’t work out. ¯\_(ツ)_/¯
      - Xylakant 8y ago
        I’d be curious what your expectations and constraints are, but from my experience of running clusters in the double digit TB-Size my ballpark figure for that amount of data would be 2 medium size data nodes and a small tiebreaker. Alternatively, if you can live with the reduced resilience and availability, even a single node might just do. Depends on the expectated churn though, ES really does not like document updates.
        FridgeSeal 8y ago
        Is that double digit TB on ElasticSearch?
        Xylakant 8y ago
        Yes, certainly.
        cinbun8 8y ago
        That does not sound like a good idea. You can't even maintain a quorum of 2 replicas with n=3 on a cluster like that. Losing one data node would be disastrous.

StavrosK 8y ago

This looks pretty cool! I'd like to try it, but it looks like I'll have to wait for the Python client library first.

deadwisdom 8y ago
Wanna build it with me?
- arcticbull 8y ago
  The Rust Python bindings are actually pretty good, I hacked together a project to let you deploy Rust micro services in Lambda via Python module bindings a while back.
- StavrosK 8y ago
  Normally I would, but I don't have an immediate use for this, which kills my motivation
marmaduke 8y ago
The protocol description makes it look fairly trivial to script, https://github.com/valeriansaliou/sonic/blob/master/PROTOCOL.md https://github.com/valeriansaliou/sonic/blob/master/PROTOCOL...
21stio 8y ago
a grpc endpoint would be nice
MosheZada 8y ago
I just wrote a basic one https://github.com/moshe/asonic https://github.com/moshe/asonic
- StavrosK 8y ago
  That looks great, thank you! I'll try it out and send you any feedback I have. Also, you might want to link to Sonic in the README, for people unfamiliar with it.

mrec 8y ago

Huh. When this was initially posted it had a weird and commercially restrictive license, but it looks like that's been reverted, possibly after (polite) discussion on /r/rust. It's MPL 2 now. https://github.com/valeriansaliou/sonic/issues/52#issuecomment-475911660 https://github.com/valeriansaliou/sonic/issues/52#issuecomme...

sidcool 8y ago

I would like to understand the code for the project. What approach would help?

d33 8y ago
Personally, I would advise finding an itch to scratch - something you'd like to see improved. Then try to understand the code from this perspective - where would you put the functionality, which pieces would it be connected to? Try to follow the lead, make notes as you read the code. You'll eventually get a feel of the infrastructure, going from the deal to the big picture - this is my default way of navigating projects. If at any moment you feel lost, look through related issues and merge requests, as well as Git history. Perhaps you'll see how things get changed in the project, patterns intrinsic to it. Also, keep in mind that you can always try to contact the community/author or invite someone to try figuring out your goal with you - once you collaborate, you'll get a solution tailored to the way you think. It does engage other people, but it also makes coding social and (at least to me) more satisfying. Let me know what you think of this approach!

mellab 8y ago

Wow. I’ve spent the last three weeks building a custom search solution in Kotlin - in my case I’m using tokenizers from Lucene and using a radix trie as an index. I actually looked at using Bleve (another rust search lib) initially but it didn’t have the right language support Glancing over this it looks like a nearly perfect fit for my use case I just wish I had seen this a couple of weeks earlier!

O_H_E 8y ago
And that's why I believe that the search/discovery problem is not solved yet by google
- StavrosK 8y ago
  Far from it, search sucks nowadays. Just think about how hard it is to find stuff from Twitter, reddit etc, even though that's where most of the content is. And if you don't know what you want exactly, you can't find anything.
- manigandham 8y ago
  Google is a public service to search the internet. This product and thread is more about adding search functionality to other applications and private data.
networkimprov 8y ago
Bleve, from Couchbase, is in Go :-) https://github.com/blevesearch/bleve https://github.com/blevesearch/bleve

marmaduke 8y ago

This looks like a breath of fresh air. Elasticsearch won’t even start if it can’t preallocate 2 GB

Xylakant 8y ago
That’s just false. The default config might set the JVM HEAP to 2GB (though I’m fairly certain it’s 1GB) but ES will start up with half a GB of heap with no issue.
- marmaduke 8y ago
  It was the default behavior on my install, documented recommendation is half the system memory and the logs don’t provide useful info when the heap allocation fails. Perhaps the upstream default is 1 GB but this was not the default, and there is much confusion on setting these correctly https://stackoverflow.com/a/40333263 https://stackoverflow.com/a/40333263 Not everyone has time to dig into it; of course if you did that’s good for you
  - Xylakant 8y ago
    What your system install does and whether the default is good for what you intend to do is a bit different from what the system allows. If you want to use a piece of software, you’re expected to somewhat understand what it does. At least reading the “setup” chapter of the documentation which explains which ENV var to set should be recommended. (I specifically link to the 1.7 docs to show that it’s been in there for quite some time) https://www.elastic.co/guide/en/elasticsearch/reference/1.7/setup-configuration.html https://www.elastic.co/guide/en/elasticsearch/reference/1.7/... The recommendation is btw: at most half of the systems memory, no more than 31g (due to how the jvm compresses heap pointers), and keep at least as much memory available for disk buffers/caches as you allocate to the heap.

javitury 8y ago

Performance figures are awesome. Also language support is great. > Sonic only keeps the N most recently pushed results for a given word This index discards old entries. This is fine for messages, in which aging items lose relevance. Yet the developer uses it for a help desk, which I think should give equal importance to all items. In this area I would say the main comperitor is Groonga. It can be integrated into PostgreSQL with the PGroonga extension, and it indexes all of the data. However it consumes way more ram.

manigandham 8y ago

Always nice to see more alternatives to Elasticsearch. That project could be so much better with some proper planning and focus. There's also Toshi: https://github.com/toshi-search/Toshi https://github.com/toshi-search/Toshi which is built on top of Tantivy: https://github.com/tantivy-search/tantivy https://github.com/tantivy-search/tantivy And for C++, there's Xapiland: https://github.com/Kronuz/Xapiand https://github.com/Kronuz/Xapiand And for Go, there's Blast: https://github.com/mosuka/blast https://github.com/mosuka/blast built on Bleve: https://github.com/blevesearch/bleve https://github.com/blevesearch/bleve