4 ms·

For anyone who wanted out of this fiasco, checkout MeiliSearch: https://docs.meilisearch.com/ https://docs.meilisearch.com/ It's written in asynchronous Rust w

by stevefan1999 5y ago

For anyone who wanted out of this fiasco, checkout MeiliSearch: https://docs.meilisearch.com/ https://docs.meilisearch.com/ It's written in asynchronous Rust with native application speed albeit using much lower memory usage than ElasticSearch, and comes with even more features than ES, so it's feature-rich, blazing fast and can still benefit on multithreaded CPUs. Downside is that it does not have distributed indexing mode yet, but it is scheduled on this year (presumably Q4 2022 I guess)

ramoz 5y ago
Milisearch is cool, but not a solution for “anyone” working on complex & large scale search.
jillesvangurp 5y ago
I'm sure it is fine and has interesting features. But at first glance it's objectively a tiny subset of the features that either Elasticsearch or Opensearch has. If that solves your problem; great. But if you need a good all round solution for search I would not start with this. The low end of the market is well covered with solutions that don't really offer a lot of features, are a bit challenged on the scalability front, lack usability, etc. Some of those have more merit than others of course than others. Things are not automatically better when they are half implemented in Rust. Lucene is an amazing piece of technology that over the years has resisted multiple attempts for people to do better in other languages. Contrary to the popular belief, it's actually pretty good with memory. Most of it is off heap memory or operating system file caches these days: it relies on memory mapped files for a lot of things. It's also pretty good with doing things concurrently. E.g. updating index files with 32 CPU cores while also serving queries is not an easy problem to solve. I've indexed documents at a rate of 500M/second on a 30 node cluster once. That's pretty amazing to see happening. I was basically saturating IO and CPUs. Indexed over a billion documents in about 1 hour. Lucene has many issues; scaling isn't one of them.

ramoz 5y ago

Milisearch is cool, but not a solution for “anyone” working on complex & large scale search.

jillesvangurp 5y ago

I'm sure it is fine and has interesting features. But at first glance it's objectively a tiny subset of the features that either Elasticsearch or Opensearch has. If that solves your problem; great. But if you need a good all round solution for search I would not start with this. The low end of the market is well covered with solutions that don't really offer a lot of features, are a bit challenged on the scalability front, lack usability, etc. Some of those have more merit than others of course than others. Things are not automatically better when they are half implemented in Rust. Lucene is an amazing piece of technology that over the years has resisted multiple attempts for people to do better in other languages. Contrary to the popular belief, it's actually pretty good with memory. Most of it is off heap memory or operating system file caches these days: it relies on memory mapped files for a lot of things. It's also pretty good with doing things concurrently. E.g. updating index files with 32 CPU cores while also serving queries is not an easy problem to solve. I've indexed documents at a rate of 500M/second on a 30 node cluster once. That's pretty amazing to see happening. I was basically saturating IO and CPUs. Indexed over a billion documents in about 1 hour. Lucene has many issues; scaling isn't one of them.