14 ms·

Sonic: Fast, lightweight and schema-less search backend

575 points by rcarmo 4y ago

dang 4y ago
Related: Sonic: Fast, lightweight and schemaless search back end in Rust - https://news.ycombinator.com/item?id=19471471 https://news.ycombinator.com/item?id=19471471 - March 2019 (39 comments)
ggeorgovassilis 4y ago
I don't see the use case for Sonic. It doesn't support clusters, which is where ES really shines, and if footprint for a local instance is a concern I'd go for SQLite.
giancarlostoro 4y ago
Now if it were drop-in capable and still more efficient, that would be impressive and I would count the days until Elastic buys you out.
eerikkivistik 4y ago
About 2 weeks ago, I was searching for an alternative to Elastic for this exact use case. Funny how the world works, now I have my answer: "someone has built it".
mhitza 4y ago
The readme doesn't offer enough information to accept that it can be an alternative to elasticsearch. From what I can gather by skimming the information, it can only do word level matching and that it isn't some form of TF-IDF type index (as is Lucene, which stands behind Solr/ElasticSearch).
- sanxiyn 4y ago
  Yes, it doesn't do any ranking at all. Results are returned in the reverse order of indexing.
speps 4y ago
Does anyone know of an alternative for the time series side of Elastic?
- snikolaev 4y ago
  Manticore Search. Here's a blog post with detailed comparison [1] [1] https://manticoresearch.com/blog/manticore-alternative-to-elasticsearch/ https://manticoresearch.com/blog/manticore-alternative-to-el...
- gkorland 4y ago
  You might want to check Redis-Stack - https://redis.io/docs/stack https://redis.io/docs/stack. It's a stack on top of Redis, which come bundled with RedisTimeSeries, RediSearch, and RedisJSON (also includes RedisGraph and RedisBloom).
nathell 4y ago
I've written a full-text search engine as well. I don't tout it as a replacement for Elasticsearch, but it does have a few advantages: it's fast; supports HTML documents; supports Polish inflection (via a full-blown morphological dictionary, not just a stemmer); and has a very compact on-disk format (pre-parsed HTML trees, Huffman-encoded over large alphabets). Oh, and it's 100% Clojure. It underlies a concordancer GUI called Smyrna: https://github.com/nathell/smyrna https://github.com/nathell/smyrna, https://smyrna.danieljanus.pl https://smyrna.danieljanus.pl I haven't touched it in six years, other than a few small changes. But I do plan on revisiting it when time permits.
- johnebgd 4y ago
  That’s very cool. I hope you consider open sourcing it so others can contribute.
  - nathell 4y ago
    It is open-source already (MIT)! I just need to make other languages more easily pluggable, and factor out the search engine so that it can be used on its own. :)
- _tom_ 4y ago
  Could your steamer be ported to Lucene? Might get more usage there.
excsn 4y ago
This is not a direct alternative to ElasticSearch. Tantivy is closer to an alternative to ElasticSearch since ES is built on top of Lucene. An alternative could be achieved if built on top of Tantivy. Sonic here only returns document identifiers so you will never be able to get document information back. This is very useful though if all you want to do is index text data and then get the stored information from another data store.
- codedokode 4y ago
  > Sonic here only returns document identifiers In many cases that is what you want because you have the data in a database and don't want to duplicate it in Elastisearch.
- counttheforks 4y ago
  > Sonic here only returns document identifiers so you will never be able to get document information back Why would you want that anyway? Always thought it was silly to duplicate all your data which will be stored in a real database anyway
  - excsn 4y ago
    From a use case I am not experienced with. If you index books, you want the search engine to return highlighted data like google does. Also, now that I think of it, typically logs/structured data is stored only in ES.
- sanxiyn 4y ago
  Quickwit is a search engine built on top of Tantivy (by the author of Tantivy): https://github.com/quickwit-oss/quickwit https://github.com/quickwit-oss/quickwit Quickwit supports Elasticsearch compatible bulk indexing API.
pavelevst 4y ago
It would be nice if it can be replacement for logging stack, elastic is super hungry for ram
- IceWreck 4y ago
  Look at zincsearch. Its another lightweight elastic alternative and they advertise logging as a usecase.
- francoismassot 4y ago
  You can have a look at Quickwit (https://quickwit.io https://quickwit.io), it's a search engine made for logs :). It's still pretty young and... there are way less features than in ES. (disclaimer: I'm one of the cofounder)
  - GordonS 4y ago
    I don't see any mention on the website on how it compares to Loki, which is surprising since it looks like a very similar model?
marginalia_nu 4y ago
* We imported ~1,000,000 messages of dynamic length (some very long, eg. emails); * Once imported, the search index weights 20MB (KV) + 1.4MB (FST) on disk; This is almost unbelievably succinct! If you encode the document features into 8 bits per document, and thus completely forego the need to store the document ID by indexing them implicitly, that alone is 1 MB. Getting meaningful search out of on average 21 bytes per document seriously impressive. [For reference, this sentence is 42 bytes.]
- mattb314 4y ago
  Wonder if this has anything to do with the sliding window: > Sonic only keeps the N most recently pushed results for a given word, in a sliding window way (the sliding window width can be configured) Default window looks like 1k documents. I read this as saying that super common words are basically dropped from the index (only 1k out of many thousands of docs retained), but I don’t know enough about the internals to be sure. Not sure if this actually hurts search results in practice, seems like an ok trade off for help docs at least.
  - 411111111111111 4y ago
    It's definitely a great trade-off to make for efficiently, but makes it inherently unusable for most of elastic searchs usecases. Looking at it from a practical example such as log search (almost everyone I know has used kibana/logstash/elasticsearch at some point): you'd be able to search for things like tracingId/requestId but adding more filters such as logLevel, requestType or serviceName would be impossible It has it's niche, but calling it an elasticsearch alternative really is a stretch
    - rabuse 4y ago
      Also the ability to weight fields when fetching results to boost relevancy, which is needed for a lot of my use cases.
  - nightpool 4y ago
    I wonder how easy it would be to change "most recently pushed" to something like a redis sorted set where each document has a score and only the top N results are retained when sorted by their separate score value? That would allow you to sort by pageviews / popularity in a more useful way. But it fails entirely when looking for uncommon intersections of common words, which feels like it makes it useless for most actual full-text search use-cases :(
syrusakbary 4y ago
Long ago I was searching on lightweight search engines that could run on the Edge, as ElasticSearch –while very popular– is also quite heavy and relies on the Lucene/JVM. Apart from Sonic, I also found Tantivy [1] and Meilisearch [2]... all delightfully made in Rust. My favorite, and the closest one to ElasticSearch (for its features) is probably Tantivy. I'd recommend anyone to check up this three projects and choose on what best fits your needs... it's awesome to see that more projects are becoming available by the day! [1]: https://github.com/quickwit-oss/tantivy https://github.com/quickwit-oss/tantivy [2]: https://github.com/meilisearch/meilisearch https://github.com/meilisearch/meilisearch
- codedokode 4y ago
  There is also sphinx search which was open source before 3.0 version.
  - snikolaev 4y ago
    And it's open source continuation - Manticore Search [1] [1] https://manticoresearch.com/ https://manticoresearch.com/
- croes 4y ago
  Do they support document access control like ES does?
  - sanxiyn 4y ago
    Yes, Meilisearch supports ES-like document access control.
- alserio 4y ago
  I've looked up tantivy and quickwit. Quickwit uses tantivy as the engine. It has decoupled storage (awesome, only recently elastic announced something comparable) but is oriented towards log processing and esplitly warns against its use to power an user facing site search. Do you happen to know if there's anything like that with the same minimal footprint that can scale up and, importantly, down to serve the needs of highly variable traffic websites? Right now I'm looking at something with clustering capabilities and decoupled storage (e.g on s3) like quickwit
dewey 4y ago
Another interesting alternative: https://github.com/meilisearch/meilisearch https://github.com/meilisearch/meilisearch - I'm using it in one of my (small) projects and I had a good experience with it, also very helpful community.
PedroBatista 4y ago
While I get the wants-and-needs since ElasticSearch has a voracious appetite for RAM, I get the feeling most people think search engines are a simple thing where you can just import some lib, fool around for a bit and call it a day. The truth is that ElasticSearch/Solr/Lucene is orders of magnitude more complex and powerful than these "alternatives". All this is mostly fine as long everyone is on the same page regarding the expectations. Most people don't need ElasticSearch for their use cases on the surface, but I feel they expect top-notch mind-reading results and that requires something like ElasticSearch and someone who knows the field. Having said all of that, Meilisearch and this are quite fine.
- alessmar 4y ago
  I would like to suggest https://typesense.org/ https://typesense.org/ It has some features that makes it a better choice than Meilisearch
  - paraboul 4y ago
    Can you elaborate on said features? I migrated from typesense to Meilisearch on a project after I found it had much better search accuracy. I can't exactly explain why, but overall Meilisearch results feel more relevant by default.
    - jabo 4y ago
      I work on Typesense. Mind if I ask which version of Typesense and Meilisearch you tried this on? And if this was on some public dataset I can use? I’d love to take a closer look.
      - paraboul 4y ago
        Hey jabo, I migrated in April 2021 (latest version of typesense & meilisearch at that time). I don't have a public dataset has it was a fairly large ecommerce catalog with close to ~500k entries. And again, it was just my own perception which is hard to define. I just found that Typesense was a bit off compared to Meilisearch on search accuracy, and of course could totally be different today with a more recent release.
codedokode 4y ago
I am not sure if it can be called an "alternative". ElasticSearch has thousands of features and settings while this library seems to be just a simple inverted index implementation only for text search. By the way if you are looking for lightweight "alternative" for ElasticSearch you might look at sphinx search engine (although it doesn't has as much features as ES has and it has became closed-source since 3.0 version).
- snikolaev 4y ago
  > you might look at sphinx search engine Manticore Search [1] forked from the latest open source version and has been continually improved for more than 5 years. [1] https://manticoresearch.com/ https://manticoresearch.com/ > although it doesn't has as much features as ES has Manticore unlike Sphinx is much closer to Elasticsearch in terms of features set.
9dev 4y ago
Every time someone comes up with an alternative to a software behemoth like Elasticsearch, what they actually mean is: "An alternative to the 10% of functionality of $tool that are interesting to me". This is surely an impressive engineering feat, but hardly a replacement for the myriad of query possibilities Elasticsearch offers.
- marginalia_nu 4y ago
  To be fair, there is often a better reasons to only replace a portion of ES' functionality, since doing so can save a lot of computation and space; than to replace ES itself, since it already exists and does a good job if what you need is the full kit. I found myself last week reimplementing 10% of RoaringBitmap's functionality as a homebrew replacement, because doing so was 500% faster. Not that RB isn't great, but it's designed for a general problem space, and not my particular problem.
  - unrealhoang 4y ago
    can you explain more of your problem and solution? Or is your solution open/published anywhere?
- jamil7 4y ago
  Agreed, although to be fair to the actual author (assuming they didn't post this here) the readme is a lot more upfront about it's capabilities. > Sonic can be used as a simple alternative to super-heavy and full-featured search backends such as Elasticsearch in some use-cases.
- RicoElectrico 4y ago
  Honestly most of the "alternative to" programs do not meet expectations they set by dropping a big known name. So much so, that I think people are doing FOSS disservice by comparing to those who they can't meaningfully overtake. The only exceptions could be small single feature utilities.
  - graftak 4y ago
atesti 4y ago
>Also, Sonic only keeps the N most recently pushed results for a given word, in a sliding window way (the sliding window width can be configured) Does this mean that it only ever finds at most N documents per word? Even searches for "A and B" would probably not find everything, even if less than N documents contain A and B, because they might have been removed with the sliding window already for A or B alone. Is that correct?
- sanxiyn 4y ago
  As far as I can tell, yes, this is correct.
- Aeolun 4y ago
  Huh? Yeah. I can keep my index size down by throwing results away as well. Every time you think it’s somehow magic, someone has to dump a bucket of cold water over your head.
hardwaresofton 4y ago
Wow it's weird that this comes up, I'm actually running a site I am going to repost to HN today that I want to use as a testbed for search engines (kind of like an extension to my recent collaboration with supabase[0]). Right now I've got the site going on just Postgres FTS + trigram and it's pretty darn fast, looks like I need to test sonic too. Going to burn some midnight oil (in my timezone, anyway) and get it out -- though sonic isn't implemented yet! Anyway to make this comment useful to people, here's my short list of engines that I want to run in parallel: - MeiliSearch (https://github.com/meilisearch/MeiliSearch https://github.com/meilisearch/MeiliSearch) - TypeSense (https://github.com/typesense/typesense https://github.com/typesense/typesense) - Lyra (https://github.com/LyraSearch/lyra https://github.com/LyraSearch/lyra) - OpenSearch (https://github.com/opensearch-project/OpenSearch https://github.com/opensearch-project/OpenSearch) - ZincSearch (https://github.com/prabhatsharma/zinc https://github.com/prabhatsharma/zinc) - Sonic (https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic) There isn't enough out there comparing all these for the simple typical fuzzy search/search box usecase, so I'm adapting a little podcast search site I made to try and use all of these at the same time. So far only Postgres though, will try and add Meilisearch today and post it! Like other people are pointing out, most of these engines won't have all the features of ES (or more accurately Lucene) but I am pretty convinced that most of the time it doesn't actually matter and if someone is searching on your site excessively maybe there's a problem with your UX (unless you're a search engine or repository of information). [0]: https://supabase.com/blog/postgres-full-text-search-vs-the-rest https://supabase.com/blog/postgres-full-text-search-vs-the-r...
- hawski 4y ago
  Thank you for this comparison. I would also like to know how Bleve Search (https://github.com/blevesearch/bleve https://github.com/blevesearch/bleve) turns out. I have for many years now a small search engine project in my free-time pipeline, but I'm before crawling even and I intend to sit for searching part after some of that.
  - hardwaresofton 4y ago
    You're right I should put bleve on there as well. This isn't even the whole list. Toshi (https://github.com/toshi-search/Toshi https://github.com/toshi-search/Toshi) is also out there...
thedougd 4y ago
Just another plug for Lucene or the library route. I had a simple use case to offer a search/autocomplete API for the employee directory of ~50,000 records. The source of truth was only updated once a day. We ran a job that reindexed daily and published the index as a file (< 15 megabytes) to where the service could access it. That service worked beautifully. Results were returned in 10-20ms and we only ever made software updates to handle the occasional CVE. It did, however, take quite a bit of fiddling initially to get the query results to match the user expectations. For example, weighting first vs last vs full name.
Thaxll 4y ago
So I can use that to inject millions of logs daily and it will do sharding and rebalancing automatically?
- sanxiyn 4y ago
  No. Sonic is a single node server and not distributed.
endisneigh 4y ago
I wish someone would write a full text engine that supports pluggable storage engines.
- ilyt 4y ago
blacklight 4y ago
While I really like their lightweight, SQL-like protocol instead of Elasticsearch's fat JSON, I really think that this project could have much more impact if it could be a drop-in replacement for ES. Even if it offers only a fraction of the features offered by ES, that may be fair enough for at least half of the use-cases out there. Sonic could have really had a strong selling point: "Use an ES-alternative that works fine in most of the real-world applications, but it's written in Rust and it only takes a fraction of the memory footprint required by ES, and it shouldn't require you to change your application code". Instead, they are proposing yet another search protocol, that developers have to learn and adopt. That definitely increases the adoption barriers.
- tensor 4y ago
  It's probably fairly easy to write an adapter here.
- markandrewj 4y ago
  Although not exactly the same, Elastic has an SQL query syntax which can be used now as well. https://www.elastic.co/what-is/elasticsearch-sql https://www.elastic.co/what-is/elasticsearch-sql
  - _boffin_ 4y ago
    They’ve had it for a long time already. Was using it over a year and a half ago. Not really “new”
    - neodymiumphish 4y ago
      In fairness, the comment you're replying to didn't say it was new.
- xvello 4y ago
  Since Elastic spitefully patched all of their client libraries to fail if the server is not a "genuine" ES server, I don't see what good a drop-in replacement with protocol compatibility would do. Go client: https://github.com/elastic/go-elasticsearch/blob/3985f2a15546ec5ed153ee696601d2a7c4e35ecc/elasticsearch.go#L322-L328 https://github.com/elastic/go-elasticsearch/blob/3985f2a1554... Python client: https://github.com/elastic/elasticsearch-py/commit/e72aa3e248fd4da95dfd8e2efd87fc6bc00ccd93 https://github.com/elastic/elasticsearch-py/commit/e72aa3e24...
daitangio 4y ago
Nice. I have done some tests with SQLlite, and I find its index module very interesting, also because it offers stemming, which seems missed here: am I wrong? SQLite has stemming only for english out-of-the-box, but I find it quite a need for a good ES drop in replacement. My two cents
- rcarmo 4y ago
  It is great and works, but sonic has broader applications (I found it because it was actually being used as a way to index an existing SQLite database that pointed to file storage).
scottwick 4y ago
Does anyone have any recommendations of books or other resources that go over the theory behind full-text search? i.e. language processing, data encoding, on-disk storage and retrieval, etc.
- snikolaev 4y ago
  https://nlp.stanford.edu/IR-book/information-retrieval-book.html https://nlp.stanford.edu/IR-book/information-retrieval-book....
- sanxiyn 4y ago
  If you want a book, Managing Gigabytes is still pretty good.
erikcw 4y ago
One of the features I like in ES that I haven’t seen in alternatives is “Percolate queries” (queries where you feed the service a document and it returns a list of queries that you’ve indexed that would match that document - basically inverting the whole process). Does anyone know of any alternatives that support this use case? https://www.elastic.co/guide/en/elasticsearch/reference/master/query-dsl-percolate-query.html#query-dsl-percolate-query https://www.elastic.co/guide/en/elasticsearch/reference/mast...
- snikolaev 4y ago
  Yes. Manticore Search does. Here's an interactive course[1] about it, it's a little bit outdated though. More info in the docs[2] [1] https://play.manticoresearch.com/pq https://play.manticoresearch.com/pq [2] https://manual.manticoresearch.com/Creating_an_index/Local_indexes/Percolate_index https://manual.manticoresearch.com/Creating_an_index/Local_i...
AndrewKemendo 4y ago
If anyone has been successful compiling this with VSCode on Win10 please let me know how you get CLANG/LLVM to play nicely with VSCode. I'd like to avoid compiling LLVM from source if I can
croes 4y ago
Most of the time these ES replacements lack a decent access control. One thing is to find what you search, but the other is not to find what you aren't allowed to see.
- sanxiyn 4y ago
  Meilisearch supports ES-like document access control.
eric4smith 4y ago
What about relevancy? There’s not much mention of that. I’m always on the lookout for something lightweight that improves on PostgreSQL full text.
- sanxiyn 4y ago
  Sonic doesn't do any ranking other than latest first.
  - eric4smith 4y ago
    Ouch
IYasha 4y ago
But does it scale?
- sanxiyn 4y ago
  No, it doesn't.
manigandham 4y ago
Lots of (elastic)search alternatives now, I keep track here: https://gist.github.com/manigandham/58320ddb24fed654b57b4ba22aceae25 https://gist.github.com/manigandham/58320ddb24fed654b57b4ba2... Sonic is good. Typesense is probably what most are looking for as more of an Algolia-like setup: https://typesense.org/ https://typesense.org/
marsven_422 4y ago
keroro 4y ago
There's also mellisearch which is another elasticsearch alternative written in rust. Comparison to elasticsearch: https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html https://docs.meilisearch.com/learn/what_is_meilisearch/compa... Github: https://github.com/meilisearch/meilisearch https://github.com/meilisearch/meilisearch Website: https://www.meilisearch.com/ https://www.meilisearch.com/
habibur 4y ago
First thing I looked for is how long does it takes to delete a document from the index. Looks like it rebuilds the whole index periodically and that's very processor intensive. The delete will be reflected after a rebuild.
pipeline_peak 4y ago
If they keep introducing hipster names like Deno and Sonic, no one will know what anything means anymore.
DeathArrow 4y ago
>Also, Sonic only keeps the N most recently pushed results for a given word, in a sliding window way (the sliding window width can be configured) If you discard many potential hits, why not use /dev/null as the search engine?
- nine_k 4y ago
  This thing looks like a very genetic cache. You can of course use /dev/null as a degenerate cache, without any performance benefit though.
- Someone1234 4y ago
  I believe you must have misread what you quoted, because whatever point you're trying doesn't really follow what you quoted. They let you configure the number of expected results to cache for a given query, the number of cache results are configurable based on your use-case for the results (e.g. if your website only lists 100 results, don't store beyond that). If more results than that for a given query are returned then they disregard additional results since you told it you won't make use of them. In essence, they're saving you from caching results that you'll never consume. How you got from this to "just use /dev/null" is a mystery to me. It has to be a misread or misunderstanding.
cies 4y ago
Other: https://www.meilisearch.com/ https://www.meilisearch.com/ https://github.com/quickwit-oss/tantivy https://github.com/quickwit-oss/tantivy https://github.com/toshi-search/Toshi https://github.com/toshi-search/Toshi https://github.com/typesense/typesense https://github.com/typesense/typesense
vlovich123 4y ago
Using a 32 bit ID is an interesting choice. It means you can only index 64-bits per bucket. I wonder if using a varint encoding would give you even more savings while handling > 4 billion documents at the cost of a bit more expensive serialization/deserialization cost (which should be negligible in the grand scheme of everything else being done).
didip 4y ago
Somewhat related, this guy: https://github.com/mosuka/ https://github.com/mosuka/ seems to be very passionate about search service. He built two distributed search services: - https://github.com/mosuka/phalanx https://github.com/mosuka/phalanx, written in Go. - https://github.com/mosuka/bayard https://github.com/mosuka/bayard, written in Rust.
airstrike 4y ago
IANAL but it seems to me like using the name "sonic" and a hedgehog mascot together teeters on the edge of IP infringement
Thaxll 4y ago
Is Sonic clustered? otherwise it's useless There are already millions of solution to have entries in an inverted index that you can query within ms, none of them have the power of ES in term of features, scaling and HA.
hxl 4y ago
A slight name-conflict with sanic: https://github.com/sanic-org/sanic https://github.com/sanic-org/sanic

575 points by rcarmo 4y ago

dang 4y ago

Related: Sonic: Fast, lightweight and schemaless search back end in Rust - https://news.ycombinator.com/item?id=19471471 https://news.ycombinator.com/item?id=19471471 - March 2019 (39 comments)

ggeorgovassilis 4y ago

I don't see the use case for Sonic. It doesn't support clusters, which is where ES really shines, and if footprint for a local instance is a concern I'd go for SQLite.

giancarlostoro 4y ago

Now if it were drop-in capable and still more efficient, that would be impressive and I would count the days until Elastic buys you out.

eerikkivistik 4y ago

About 2 weeks ago, I was searching for an alternative to Elastic for this exact use case. Funny how the world works, now I have my answer: "someone has built it".

mhitza 4y ago

The readme doesn't offer enough information to accept that it can be an alternative to elasticsearch. From what I can gather by skimming the information, it can only do word level matching and that it isn't some form of TF-IDF type index (as is Lucene, which stands behind Solr/ElasticSearch).

sanxiyn 4y ago
Yes, it doesn't do any ranking at all. Results are returned in the reverse order of indexing.

speps 4y ago

Does anyone know of an alternative for the time series side of Elastic?

snikolaev 4y ago
Manticore Search. Here's a blog post with detailed comparison [1] [1] https://manticoresearch.com/blog/manticore-alternative-to-elasticsearch/ https://manticoresearch.com/blog/manticore-alternative-to-el...
gkorland 4y ago
You might want to check Redis-Stack - https://redis.io/docs/stack https://redis.io/docs/stack. It's a stack on top of Redis, which come bundled with RedisTimeSeries, RediSearch, and RedisJSON (also includes RedisGraph and RedisBloom).

nathell 4y ago

I've written a full-text search engine as well. I don't tout it as a replacement for Elasticsearch, but it does have a few advantages: it's fast; supports HTML documents; supports Polish inflection (via a full-blown morphological dictionary, not just a stemmer); and has a very compact on-disk format (pre-parsed HTML trees, Huffman-encoded over large alphabets). Oh, and it's 100% Clojure. It underlies a concordancer GUI called Smyrna: https://github.com/nathell/smyrna https://github.com/nathell/smyrna, https://smyrna.danieljanus.pl https://smyrna.danieljanus.pl I haven't touched it in six years, other than a few small changes. But I do plan on revisiting it when time permits.

johnebgd 4y ago
That’s very cool. I hope you consider open sourcing it so others can contribute.
- nathell 4y ago
  It is open-source already (MIT)! I just need to make other languages more easily pluggable, and factor out the search engine so that it can be used on its own. :)
_tom_ 4y ago
Could your steamer be ported to Lucene? Might get more usage there.

excsn 4y ago

This is not a direct alternative to ElasticSearch. Tantivy is closer to an alternative to ElasticSearch since ES is built on top of Lucene. An alternative could be achieved if built on top of Tantivy. Sonic here only returns document identifiers so you will never be able to get document information back. This is very useful though if all you want to do is index text data and then get the stored information from another data store.

codedokode 4y ago
> Sonic here only returns document identifiers In many cases that is what you want because you have the data in a database and don't want to duplicate it in Elastisearch.
counttheforks 4y ago
> Sonic here only returns document identifiers so you will never be able to get document information back Why would you want that anyway? Always thought it was silly to duplicate all your data which will be stored in a real database anyway
- excsn 4y ago
  From a use case I am not experienced with. If you index books, you want the search engine to return highlighted data like google does. Also, now that I think of it, typically logs/structured data is stored only in ES.
sanxiyn 4y ago
Quickwit is a search engine built on top of Tantivy (by the author of Tantivy): https://github.com/quickwit-oss/quickwit https://github.com/quickwit-oss/quickwit Quickwit supports Elasticsearch compatible bulk indexing API.

pavelevst 4y ago

It would be nice if it can be replacement for logging stack, elastic is super hungry for ram

IceWreck 4y ago
Look at zincsearch. Its another lightweight elastic alternative and they advertise logging as a usecase.
francoismassot 4y ago
You can have a look at Quickwit (https://quickwit.io https://quickwit.io), it's a search engine made for logs :). It's still pretty young and... there are way less features than in ES. (disclaimer: I'm one of the cofounder)
- GordonS 4y ago
  I don't see any mention on the website on how it compares to Loki, which is surprising since it looks like a very similar model?

marginalia_nu 4y ago

* We imported ~1,000,000 messages of dynamic length (some very long, eg. emails); * Once imported, the search index weights 20MB (KV) + 1.4MB (FST) on disk; This is almost unbelievably succinct! If you encode the document features into 8 bits per document, and thus completely forego the need to store the document ID by indexing them implicitly, that alone is 1 MB. Getting meaningful search out of on average 21 bytes per document seriously impressive. [For reference, this sentence is 42 bytes.]

mattb314 4y ago
Wonder if this has anything to do with the sliding window: > Sonic only keeps the N most recently pushed results for a given word, in a sliding window way (the sliding window width can be configured) Default window looks like 1k documents. I read this as saying that super common words are basically dropped from the index (only 1k out of many thousands of docs retained), but I don’t know enough about the internals to be sure. Not sure if this actually hurts search results in practice, seems like an ok trade off for help docs at least.
- 411111111111111 4y ago
  It's definitely a great trade-off to make for efficiently, but makes it inherently unusable for most of elastic searchs usecases. Looking at it from a practical example such as log search (almost everyone I know has used kibana/logstash/elasticsearch at some point): you'd be able to search for things like tracingId/requestId but adding more filters such as logLevel, requestType or serviceName would be impossible It has it's niche, but calling it an elasticsearch alternative really is a stretch
  - rabuse 4y ago
    Also the ability to weight fields when fetching results to boost relevancy, which is needed for a lot of my use cases.
- nightpool 4y ago
  I wonder how easy it would be to change "most recently pushed" to something like a redis sorted set where each document has a score and only the top N results are retained when sorted by their separate score value? That would allow you to sort by pageviews / popularity in a more useful way. But it fails entirely when looking for uncommon intersections of common words, which feels like it makes it useless for most actual full-text search use-cases :(

syrusakbary 4y ago

Long ago I was searching on lightweight search engines that could run on the Edge, as ElasticSearch –while very popular– is also quite heavy and relies on the Lucene/JVM. Apart from Sonic, I also found Tantivy [1] and Meilisearch [2]... all delightfully made in Rust. My favorite, and the closest one to ElasticSearch (for its features) is probably Tantivy. I'd recommend anyone to check up this three projects and choose on what best fits your needs... it's awesome to see that more projects are becoming available by the day! [1]: https://github.com/quickwit-oss/tantivy https://github.com/quickwit-oss/tantivy [2]: https://github.com/meilisearch/meilisearch https://github.com/meilisearch/meilisearch

codedokode 4y ago
There is also sphinx search which was open source before 3.0 version.
- snikolaev 4y ago
  And it's open source continuation - Manticore Search [1] [1] https://manticoresearch.com/ https://manticoresearch.com/
croes 4y ago
Do they support document access control like ES does?
- sanxiyn 4y ago
  Yes, Meilisearch supports ES-like document access control.
alserio 4y ago
I've looked up tantivy and quickwit. Quickwit uses tantivy as the engine. It has decoupled storage (awesome, only recently elastic announced something comparable) but is oriented towards log processing and esplitly warns against its use to power an user facing site search. Do you happen to know if there's anything like that with the same minimal footprint that can scale up and, importantly, down to serve the needs of highly variable traffic websites? Right now I'm looking at something with clustering capabilities and decoupled storage (e.g on s3) like quickwit

dewey 4y ago

Another interesting alternative: https://github.com/meilisearch/meilisearch https://github.com/meilisearch/meilisearch - I'm using it in one of my (small) projects and I had a good experience with it, also very helpful community.

PedroBatista 4y ago

While I get the wants-and-needs since ElasticSearch has a voracious appetite for RAM, I get the feeling most people think search engines are a simple thing where you can just import some lib, fool around for a bit and call it a day. The truth is that ElasticSearch/Solr/Lucene is orders of magnitude more complex and powerful than these "alternatives". All this is mostly fine as long everyone is on the same page regarding the expectations. Most people don't need ElasticSearch for their use cases on the surface, but I feel they expect top-notch mind-reading results and that requires something like ElasticSearch and someone who knows the field. Having said all of that, Meilisearch and this are quite fine.

alessmar 4y ago
I would like to suggest https://typesense.org/ https://typesense.org/ It has some features that makes it a better choice than Meilisearch
- paraboul 4y ago
  Can you elaborate on said features? I migrated from typesense to Meilisearch on a project after I found it had much better search accuracy. I can't exactly explain why, but overall Meilisearch results feel more relevant by default.
  - jabo 4y ago
    I work on Typesense. Mind if I ask which version of Typesense and Meilisearch you tried this on? And if this was on some public dataset I can use? I’d love to take a closer look.
    - paraboul 4y ago
      Hey jabo, I migrated in April 2021 (latest version of typesense & meilisearch at that time). I don't have a public dataset has it was a fairly large ecommerce catalog with close to ~500k entries. And again, it was just my own perception which is hard to define. I just found that Typesense was a bit off compared to Meilisearch on search accuracy, and of course could totally be different today with a more recent release.

codedokode 4y ago

I am not sure if it can be called an "alternative". ElasticSearch has thousands of features and settings while this library seems to be just a simple inverted index implementation only for text search. By the way if you are looking for lightweight "alternative" for ElasticSearch you might look at sphinx search engine (although it doesn't has as much features as ES has and it has became closed-source since 3.0 version).

snikolaev 4y ago
> you might look at sphinx search engine Manticore Search [1] forked from the latest open source version and has been continually improved for more than 5 years. [1] https://manticoresearch.com/ https://manticoresearch.com/ > although it doesn't has as much features as ES has Manticore unlike Sphinx is much closer to Elasticsearch in terms of features set.

9dev 4y ago

Every time someone comes up with an alternative to a software behemoth like Elasticsearch, what they actually mean is: "An alternative to the 10% of functionality of $tool that are interesting to me". This is surely an impressive engineering feat, but hardly a replacement for the myriad of query possibilities Elasticsearch offers.

marginalia_nu 4y ago
To be fair, there is often a better reasons to only replace a portion of ES' functionality, since doing so can save a lot of computation and space; than to replace ES itself, since it already exists and does a good job if what you need is the full kit. I found myself last week reimplementing 10% of RoaringBitmap's functionality as a homebrew replacement, because doing so was 500% faster. Not that RB isn't great, but it's designed for a general problem space, and not my particular problem.
- unrealhoang 4y ago
  can you explain more of your problem and solution? Or is your solution open/published anywhere?
jamil7 4y ago
Agreed, although to be fair to the actual author (assuming they didn't post this here) the readme is a lot more upfront about it's capabilities. > Sonic can be used as a simple alternative to super-heavy and full-featured search backends such as Elasticsearch in some use-cases.
RicoElectrico 4y ago
Honestly most of the "alternative to" programs do not meet expectations they set by dropping a big known name. So much so, that I think people are doing FOSS disservice by comparing to those who they can't meaningfully overtake. The only exceptions could be small single feature utilities.
- graftak 4y ago

atesti 4y ago

>Also, Sonic only keeps the N most recently pushed results for a given word, in a sliding window way (the sliding window width can be configured) Does this mean that it only ever finds at most N documents per word? Even searches for "A and B" would probably not find everything, even if less than N documents contain A and B, because they might have been removed with the sliding window already for A or B alone. Is that correct?

sanxiyn 4y ago
As far as I can tell, yes, this is correct.
Aeolun 4y ago
Huh? Yeah. I can keep my index size down by throwing results away as well. Every time you think it’s somehow magic, someone has to dump a bucket of cold water over your head.

hardwaresofton 4y ago

Wow it's weird that this comes up, I'm actually running a site I am going to repost to HN today that I want to use as a testbed for search engines (kind of like an extension to my recent collaboration with supabase[0]). Right now I've got the site going on just Postgres FTS + trigram and it's pretty darn fast, looks like I need to test sonic too. Going to burn some midnight oil (in my timezone, anyway) and get it out -- though sonic isn't implemented yet! Anyway to make this comment useful to people, here's my short list of engines that I want to run in parallel: - MeiliSearch (https://github.com/meilisearch/MeiliSearch https://github.com/meilisearch/MeiliSearch) - TypeSense (https://github.com/typesense/typesense https://github.com/typesense/typesense) - Lyra (https://github.com/LyraSearch/lyra https://github.com/LyraSearch/lyra) - OpenSearch (https://github.com/opensearch-project/OpenSearch https://github.com/opensearch-project/OpenSearch) - ZincSearch (https://github.com/prabhatsharma/zinc https://github.com/prabhatsharma/zinc) - Sonic (https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic) There isn't enough out there comparing all these for the simple typical fuzzy search/search box usecase, so I'm adapting a little podcast search site I made to try and use all of these at the same time. So far only Postgres though, will try and add Meilisearch today and post it! Like other people are pointing out, most of these engines won't have all the features of ES (or more accurately Lucene) but I am pretty convinced that most of the time it doesn't actually matter and if someone is searching on your site excessively maybe there's a problem with your UX (unless you're a search engine or repository of information). [0]: https://supabase.com/blog/postgres-full-text-search-vs-the-rest https://supabase.com/blog/postgres-full-text-search-vs-the-r...

hawski 4y ago
Thank you for this comparison. I would also like to know how Bleve Search (https://github.com/blevesearch/bleve https://github.com/blevesearch/bleve) turns out. I have for many years now a small search engine project in my free-time pipeline, but I'm before crawling even and I intend to sit for searching part after some of that.
- hardwaresofton 4y ago
  You're right I should put bleve on there as well. This isn't even the whole list. Toshi (https://github.com/toshi-search/Toshi https://github.com/toshi-search/Toshi) is also out there...

thedougd 4y ago

Just another plug for Lucene or the library route. I had a simple use case to offer a search/autocomplete API for the employee directory of ~50,000 records. The source of truth was only updated once a day. We ran a job that reindexed daily and published the index as a file (< 15 megabytes) to where the service could access it. That service worked beautifully. Results were returned in 10-20ms and we only ever made software updates to handle the occasional CVE. It did, however, take quite a bit of fiddling initially to get the query results to match the user expectations. For example, weighting first vs last vs full name.

Thaxll 4y ago

So I can use that to inject millions of logs daily and it will do sharding and rebalancing automatically?

sanxiyn 4y ago
No. Sonic is a single node server and not distributed.

endisneigh 4y ago

I wish someone would write a full text engine that supports pluggable storage engines.

ilyt 4y ago

blacklight 4y ago

While I really like their lightweight, SQL-like protocol instead of Elasticsearch's fat JSON, I really think that this project could have much more impact if it could be a drop-in replacement for ES. Even if it offers only a fraction of the features offered by ES, that may be fair enough for at least half of the use-cases out there. Sonic could have really had a strong selling point: "Use an ES-alternative that works fine in most of the real-world applications, but it's written in Rust and it only takes a fraction of the memory footprint required by ES, and it shouldn't require you to change your application code". Instead, they are proposing yet another search protocol, that developers have to learn and adopt. That definitely increases the adoption barriers.

tensor 4y ago
It's probably fairly easy to write an adapter here.
markandrewj 4y ago
Although not exactly the same, Elastic has an SQL query syntax which can be used now as well. https://www.elastic.co/what-is/elasticsearch-sql https://www.elastic.co/what-is/elasticsearch-sql
- _boffin_ 4y ago
  They’ve had it for a long time already. Was using it over a year and a half ago. Not really “new”
  - neodymiumphish 4y ago
    In fairness, the comment you're replying to didn't say it was new.
xvello 4y ago
Since Elastic spitefully patched all of their client libraries to fail if the server is not a "genuine" ES server, I don't see what good a drop-in replacement with protocol compatibility would do. Go client: https://github.com/elastic/go-elasticsearch/blob/3985f2a15546ec5ed153ee696601d2a7c4e35ecc/elasticsearch.go#L322-L328 https://github.com/elastic/go-elasticsearch/blob/3985f2a1554... Python client: https://github.com/elastic/elasticsearch-py/commit/e72aa3e248fd4da95dfd8e2efd87fc6bc00ccd93 https://github.com/elastic/elasticsearch-py/commit/e72aa3e24...

daitangio 4y ago

Nice. I have done some tests with SQLlite, and I find its index module very interesting, also because it offers stemming, which seems missed here: am I wrong? SQLite has stemming only for english out-of-the-box, but I find it quite a need for a good ES drop in replacement. My two cents

rcarmo 4y ago
It is great and works, but sonic has broader applications (I found it because it was actually being used as a way to index an existing SQLite database that pointed to file storage).

scottwick 4y ago

Does anyone have any recommendations of books or other resources that go over the theory behind full-text search? i.e. language processing, data encoding, on-disk storage and retrieval, etc.

snikolaev 4y ago
https://nlp.stanford.edu/IR-book/information-retrieval-book.html https://nlp.stanford.edu/IR-book/information-retrieval-book....
sanxiyn 4y ago
If you want a book, Managing Gigabytes is still pretty good.

erikcw 4y ago

One of the features I like in ES that I haven’t seen in alternatives is “Percolate queries” (queries where you feed the service a document and it returns a list of queries that you’ve indexed that would match that document - basically inverting the whole process). Does anyone know of any alternatives that support this use case? https://www.elastic.co/guide/en/elasticsearch/reference/master/query-dsl-percolate-query.html#query-dsl-percolate-query https://www.elastic.co/guide/en/elasticsearch/reference/mast...

snikolaev 4y ago
Yes. Manticore Search does. Here's an interactive course[1] about it, it's a little bit outdated though. More info in the docs[2] [1] https://play.manticoresearch.com/pq https://play.manticoresearch.com/pq [2] https://manual.manticoresearch.com/Creating_an_index/Local_indexes/Percolate_index https://manual.manticoresearch.com/Creating_an_index/Local_i...

AndrewKemendo 4y ago

If anyone has been successful compiling this with VSCode on Win10 please let me know how you get CLANG/LLVM to play nicely with VSCode. I'd like to avoid compiling LLVM from source if I can

croes 4y ago

Most of the time these ES replacements lack a decent access control. One thing is to find what you search, but the other is not to find what you aren't allowed to see.

sanxiyn 4y ago
Meilisearch supports ES-like document access control.

eric4smith 4y ago

What about relevancy? There’s not much mention of that. I’m always on the lookout for something lightweight that improves on PostgreSQL full text.

sanxiyn 4y ago
Sonic doesn't do any ranking other than latest first.
- eric4smith 4y ago
  Ouch

IYasha 4y ago

But does it scale?

sanxiyn 4y ago
No, it doesn't.

manigandham 4y ago

Lots of (elastic)search alternatives now, I keep track here: https://gist.github.com/manigandham/58320ddb24fed654b57b4ba22aceae25 https://gist.github.com/manigandham/58320ddb24fed654b57b4ba2... Sonic is good. Typesense is probably what most are looking for as more of an Algolia-like setup: https://typesense.org/ https://typesense.org/

marsven_422 4y ago

keroro 4y ago

There's also mellisearch which is another elasticsearch alternative written in rust. Comparison to elasticsearch: https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html https://docs.meilisearch.com/learn/what_is_meilisearch/compa... Github: https://github.com/meilisearch/meilisearch https://github.com/meilisearch/meilisearch Website: https://www.meilisearch.com/ https://www.meilisearch.com/

habibur 4y ago

First thing I looked for is how long does it takes to delete a document from the index. Looks like it rebuilds the whole index periodically and that's very processor intensive. The delete will be reflected after a rebuild.

pipeline_peak 4y ago

If they keep introducing hipster names like Deno and Sonic, no one will know what anything means anymore.

DeathArrow 4y ago

>Also, Sonic only keeps the N most recently pushed results for a given word, in a sliding window way (the sliding window width can be configured) If you discard many potential hits, why not use /dev/null as the search engine?

nine_k 4y ago
This thing looks like a very genetic cache. You can of course use /dev/null as a degenerate cache, without any performance benefit though.
Someone1234 4y ago
I believe you must have misread what you quoted, because whatever point you're trying doesn't really follow what you quoted. They let you configure the number of expected results to cache for a given query, the number of cache results are configurable based on your use-case for the results (e.g. if your website only lists 100 results, don't store beyond that). If more results than that for a given query are returned then they disregard additional results since you told it you won't make use of them. In essence, they're saving you from caching results that you'll never consume. How you got from this to "just use /dev/null" is a mystery to me. It has to be a misread or misunderstanding.

cies 4y ago

Other: https://www.meilisearch.com/ https://www.meilisearch.com/ https://github.com/quickwit-oss/tantivy https://github.com/quickwit-oss/tantivy https://github.com/toshi-search/Toshi https://github.com/toshi-search/Toshi https://github.com/typesense/typesense https://github.com/typesense/typesense

vlovich123 4y ago

Using a 32 bit ID is an interesting choice. It means you can only index 64-bits per bucket. I wonder if using a varint encoding would give you even more savings while handling > 4 billion documents at the cost of a bit more expensive serialization/deserialization cost (which should be negligible in the grand scheme of everything else being done).

didip 4y ago

Somewhat related, this guy: https://github.com/mosuka/ https://github.com/mosuka/ seems to be very passionate about search service. He built two distributed search services: - https://github.com/mosuka/phalanx https://github.com/mosuka/phalanx, written in Go. - https://github.com/mosuka/bayard https://github.com/mosuka/bayard, written in Rust.

airstrike 4y ago

IANAL but it seems to me like using the name "sonic" and a hedgehog mascot together teeters on the edge of IP infringement

Thaxll 4y ago

Is Sonic clustered? otherwise it's useless There are already millions of solution to have entries in an inverted index that you can query within ms, none of them have the power of ES in term of features, scaling and HA.

hxl 4y ago

A slight name-conflict with sanic: https://github.com/sanic-org/sanic https://github.com/sanic-org/sanic