15 ms·

Meilisearch 1.0 – Open-source search engine built in Rust

448 points by tpayet 4y ago

xarope 4y ago
I tried RTFM'ing, but can Meilisearch handle restricted documents (P&C) and integrated with LDAP/AD to pull security groups? P.S. great to see your documentation search is powered by your own product (!)
fyzix 4y ago
They need to structure their pricing page better. A quick glance had me thinking that $1200 was the minimum for production use. But the free tier is actually pay as you go.
nop_slide 4y ago
This looks really cool and I might try the self hosted option out on my small website as an upgrade from Postgres’ full text search. I was hoping the cloud version would be more appealing, granted there seems to be a generous free tier but the next option is $1200 a month?!
- koblas 4y ago
  Was excited to see a non-GC'ed search engine that looked solid. But, without having the replicated - distributed version of it in the "free" tier makes it hard to really evaluate.
  - tpayet 4y ago
    Feel free to reach out to quentin@meilisearch.com, we'll find a way for you to evaluate the pro plan!
- tpayet 4y ago
  Sorry, it might not be obvious, but you can go over the free tier and pay for the usage at 0.25$ for each 1000 searches/documents :)
  - nop_slide 4y ago
    Ah yep sorry I missed that! Good to know. I just saw the next option was $1200 and my eyes became fixated on the number. Maybe I will try out the cloud version then even though I expect my site would probably be well in the free tier limit, like I said it seems like a very generous tier.
- ren_engineer 4y ago
  their free tier looks like it has a "pay as you go" option once you exceed it that's identical to the paid option per 1K searches and 1K documents. You are basically paying for priority support, pretty common strategy and seems fair to me. just noticed you don't get high availability on free tier which sucks, but I guess if search is mission critical to the point you need it, you would be willing to pay. Most of these database type companies start off targeting enterprise and then roll out self-serve solutions as they scale.
kacy 4y ago
We’ve been using a Meilisearch for the last six months or so and have been delighted with its performance and usability. It uses a fraction of the resources as Elasticsearch, and the language support is extensive and very active. That being said, our cluster is much smaller than other ones I’ve worked with in the past, so I can’t comment on its reliability at massive scale. I’ve also been very impressed with how active contributors are on GitHub and in their Discord. Everyone seems like good people, and it’s a project I’m excited to keep using.
- tpayet 4y ago
  Thank you very much! I'll share your comment with the team <3
- tempest_ 4y ago
  This is the thing I find when people post "ElasticSearch Alternative". 80% of ElasticSearch's value add (wrt search anyway) is all the clustering and frame work that allows you to span the search over tens or hundreds of machines "easily". I think the same is true here. Probably the comparison should be with the underlying search libraries that ES sits on. I suppose this comparison makes sense in a world where most people don't run their own servers much any more since the clustering etc would be a problem for the cloud offering and not the consumer.
  - Semaphor 4y ago
    > 80% of ElasticSearch's value add (wrt search anyway) is all the clustering Or configurability. I looked at this again now that 1.0 is out, but besides the .NET client still being in an alpha state, it’s also very zero-configuration. There seems to be no configurability regarding tokenization strategies, for example. Now, I certainly see the appeal, I barely understand my own ES code and meilisearch replicates probably 70% of it with no configuration at all, that’s impressive, but it also means that switching would mean giving up on those 30%.
  - kmac_ 4y ago
    Yeah, Elastic also brings advanced aggregates and filters, Kibana, nice UI where you can explore data and create dashboards easily and tons of bigger and smaller features. But in some areas both products are comparable.
manigandham 4y ago
Congrats to the team, it's been interesting to watch the development of Meilisearch (and it's close competitor Typesense). Algolia has really paved the way here but it's nice to see the open-source options with more configurations and better default UX. There's also many search libraries if you want to embed search more deeply into your app. I have a list of modern search systems and libraries here: https://manigandham.com/post/search-systems-libraries https://manigandham.com/post/search-systems-libraries
- naiv 4y ago
  I will never understand who the target group of Algolia is besides a website where the number of records coincidentally is in the range of the number of queries. At least they got rid of the pricing per indexing transaction which made it even more absurd. If Algolia would offer an instance based pricing on cpu, ram and storage they would be the clear winner imho.
  - manigandham 4y ago
    Why do the number of records and searches have to be similar? The current pricing is simple - you pay per "search unit" which scales in both dimensions. The vast majority of small/medium customers would rather pay-as-you-go than maintain a fixed cost instance, and it allows Algolia to efficiently pack them into a multitenant architecture instead of wasting resource overhead.
    - naiv 4y ago
      If you eg index geonames, you have 4 mio. records but you might only have 50.000 queries a month. you pay $4,000 for minimal compute resources, 4GB of RAM and 3 gigabytes of storage space. Would be less but algolia requires you to create a replica for each sort option separately. With 4 mio. records and 4 mio. queries I would pay the same. But then at least have 4 mio. queries. The other way around, if we would just index all 200+ countries in the world and have autocomplete with a lot of visitors we would pay for eg 50.000 users per day typing in 3 letters again $4.000. Same for us, we offer 350.000 movies with 2 mio. scenes. With Typesense or even Elasticsearch Cloud we would pay 5% of what we would pay Algolia.
chimen 4y ago
Is Rust that important that you have to place "built in Rust" in the title? Is this like a cult following that we only bet on traffic and interest coming from other evangelists where Rust is the only feature that matter? 4 months ago: " Meilisearch, open-source alternative to Algolia in Rust lands a $15M Series A" It's not the first time I see, there are at least 2-3 daily submissions reaching the FP in this manner so I'm curious: "built in Rust" = marketing these days?
- adamnemecek 4y ago
  It is important. Rust projects are infinitely easier to contribute to.
  - dividedbyzero 4y ago
    Compared to what?
    - freilanzer 4y ago
      Python?
    - bsnnkv 4y ago
      For me: compared to projects in languages with less mature tooling and compilers less capable of preventing entire classes of errors by default.
    - adamnemecek 4y ago
      Anything. Legit no language comes even close in terms of how easy it is to git clone something and get it to build.
      - hu3 4y ago
        I'd argue Go projects tend to be easier to build since they require nightly Go builds much less frequently (I don't even remember a project that ever required nightly Go tbh). https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=rust%20nightly&sort=byPopularity&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
msvan 4y ago
How does Meilisearch compare to ElasticSearch from an operational point of view? I've experienced ElasticSearch to be quite painful to maintain, requiring lots of manual tweaking to balance shards and careful design of indices.
- tpayet 4y ago
  That's the point! We don't ambition to compete with Elastic on everything (logs, analytics, etc). We are doing search for front-end users with a strong focus on relevancy, speed & developer experience. You can read a bit more on our documentation https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html#meilisearch-vs-elasticsearch https://docs.meilisearch.com/learn/what_is_meilisearch/compa...
  - sidmitra 4y ago
    A quick question, are there any limits around number of separate indexes we can have with meilisearch? I'm thinking atleast say 20-30K separate indexes to start with. My use case is that i want to start creating some indexes that are "per-user" and some "per-company" where a company(customer) might have many users. This is to do some sort of double tenant isolation. I will create different keys that have permission to specific indexes and deliver those to the user somehow. My current solution does hacky things with Elasticsearch like adding query filters by user/company-id attributes in the background automatically. But since meilisearch would be customer facing, i need stronger guarantees around permissions per index. I tried this out a year ago on Meilsearch locally, but haven't stress tested it by creating thousands of them like production. Or is there a better way to do this. This is also a reason where memory-only systems like Typesense didn't make sense to me. I'm fine with taking a performance hit by going to disk to pull the right index. Not every index will be used all the time. I might also look at sharding/partitioning features if present.
    - dureuill 4y ago
      Hello! > A quick question, are there any limits around number of separate indexes we can have with meilisearch? Yes! In v1.0, about 180 indexes under Linux in the same instance[1]. The good news is that I'm personally working on lifting this limitation for v1.1 (planned to release in the beginning of April), which should be able to accommodate an unlimited number of indexes[2] (disk space permits, of course). Note that having many indexes does have an impact on performance and will keep doing so even after v1.1. > Or is there a better way to do this. If it works for your use case, you can try using a single index (or a few indexes) with tenant tokens[3] for multitenancy. Hope this helps :-) [1]: https://docs.meilisearch.com/learn/advanced/known_limitations.html#maximum-database-size https://docs.meilisearch.com/learn/advanced/known_limitation... [2]: https://github.com/meilisearch/meilisearch/issues/3382 https://github.com/meilisearch/meilisearch/issues/3382 [3]: https://docs.meilisearch.com/learn/security/tenant_tokens.html https://docs.meilisearch.com/learn/security/tenant_tokens.ht...
drcongo 4y ago
Congrats team. Meilisearch is an absolute joy to work with.
- tpayet 4y ago
  Thanks :D
- nynapalm 4y ago
  Thanks! :)
- ferdi05 4y ago
  Thanks!
leeoniya 4y ago
compared to https://typesense.org/ https://typesense.org/ ?
- bduffany 4y ago
  typesense did their own comparison here: https://typesense.org/typesense-vs-algolia-vs-elasticsearch-vs-meilisearch/ https://typesense.org/typesense-vs-algolia-vs-elasticsearch-...
  - deleted 4y ago
    [deleted]
  - curquiza 4y ago
    Unfortunately, the comparison with Meilisearch is not up to date in this link. Also, we have to keep in mind that every comparison written by a company is always oriented.
    - jabo 4y ago
      I maintain that comparison page on the Typesense side. I just updated it as recently as yesterday, based on my observation. But let me know which ones need updating for Meilisearch. Happy to update. While we’re on the topic, reminder about some of the outdated information in your comparison pages: https://twitter.com/typesense/status/1620825236055932928?s=46&t=ryPI1ZnQYEvrXgPmPl9I5Q https://twitter.com/typesense/status/1620825236055932928?s=4...
      - traverseda 4y ago
        I'd say that the bit where typesense can only work with data that fits in ram is actually a pretty big problem for a lot of use cases, as an aside. That feature alone would discount typesense for basically all of my personal projects. Might be a trade off I'd be willing to make on a professional project given the other features but it seems really wasteful. Personally I find the meilisearch comparison to be more useful for the type of stuff I'm doing: https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html https://docs.meilisearch.com/learn/what_is_meilisearch/compa... Of course I'm not a large enterprise e-commerce site. I'm doing personal projects like web archiving, (dataset probably won't be anywhere near fitting in ram) or I'm using search engines on embedded devices (search needs to play well with others, not use all my ram).
tmikaeld 4y ago
My team tried to use Meilisearch for large datasets, unfortunately, it's impossible to plan the RAM usage. If you have very little searches, it consumed very little, but if you have a lot of search traffic, it may consume more than we could provision beforehand. This made it too unpredictable and too expensive, so we went with Manticore instead. I don't know if this has been addressed in 1.0, hopefully it has.
- marban 4y ago
  Do you have any numeric definitions for few and lots?
  - traverseda 4y ago
    I think that they might have fixed it. I noted this as a problem with earlier meilisearch releases as well, but reading through the documentation it looks like they don't require the entire index to be in memory any more, allowing it to be a memory mapped file. https://docs.meilisearch.com/learn/advanced/storage.html#lmdb https://docs.meilisearch.com/learn/advanced/storage.html#lmd... >For the best performance, it is recommended to provide the same amount of RAM as the size the database takes on disk, so all the data structures can fit in memory. > [...] >It is important to note that there is no reliable way to predict the final size of a database. This is true for just about any search engine on the market—we're just the only ones saying it out loud. Looks like a 10MB document is taking ~200MB, from their docs. I don't think that scales linearly though, since it's a reverse index it is going to scale based on the number of unique words it finds, with each document adding a bit on top of that. You'd expect it to have a pretty big index to cover common english words, and then each document adds a bit on top of that. Definitely seems like somewhere they could make some improvements though. Some transparent compression could probably help, and with zstd's dictionary feature it can be fine tuned to the data they're actually seeing. Not about to replace xapian in kiwix (offline wikipedia reader) any time soon, I think.
  - tmikaeld 4y ago
    Our index was aimed at handling 20 000 documents at total of 35MB of CSV, this would balloon into 0.7GB to 1GB of RAM and we expected at least 1000 of these indexes, which would require dedicated servers with 1TB of RAM. This was when Meili was at version 0.27. With manticore, we've tried to run into these issues in benchmarks, but the only problem we got was temporary high IO load when indexes need to be re-indexed with new or changed documents. In total it's at 50-70% of the RAM usage compared to Meili. We'd be happy to re-visit, but looking at the docs - it seems to be about the same as it was back then (a year ago).
kristiandupont 4y ago
I am using the core (called "Milli") in a local indexer that I run on my repositories and Obsidian files. It works like a charm and I am very happy with it. Obviously that's a use case with very little traffic but just indexing my repositories folder is quite a bit of work and it does it surprisingly fast. The only real thing I am missing is a typeahead feature.
- dureuill 4y ago
  Hello from a Meilisearch team member, wow your project looks very interesting. How do you handle things like the filesystem changing while your indexer is offline? Do you reindex from scratch at startup? Regarding typeahead, is this what we call "query suggestions"[1]? At the moment, we think that this is something that frontends and SDK can provide rather than the engine, so that means you wouldn't find it at the Milli level. We think you could maybe build an ancillary suggestion index and make two queries instead of one when typing, so as to get both results and suggestions at once. Here's a chat link[2] to our latest discussions on the topic; feel free to come and weigh in if you're interested! [1]: https://roadmap.meilisearch.com/c/31-query-suggestions https://roadmap.meilisearch.com/c/31-query-suggestions [2]: https://discord.com/channels/1006923006964154428/1068507365801992272/1068507369895633056 https://discord.com/channels/1006923006964154428/10685073658...
  - kristiandupont 4y ago
    Thank you! Yes, I reindex. I store the file timestamp along with the contents, so it's not quite as involved as it could seem but startup does take a bit. And, I don't have a good way of discovering deleted files at the moment. Not a big deal as it is, but something I will look into. And yes, query suggestions are exactly what I mean. Thank you for informing me, I guess I will have to look into how I can make it myself :-)
    - dureuill 4y ago
      You could maybe use something equivalent to the "index hot swap"[1] feature we have at the Meilisearch level at startup, so that you make the reindexing in a another index at startup, and then atomatically swap this fresh index with the old one when it is ready? That way, you have fast startup at the cost of having possibly out-of-date information for a while after startup. (you could even reindex from scratch completely in the background at startup, so no need to discover deleted files at all) [1]: https://blog.meilisearch.com/zero-downtime-index-deployment/ https://blog.meilisearch.com/zero-downtime-index-deployment/
snowpid 4y ago
Can you rewrite it in Rust?
- curquiza 4y ago
  We will think about it :D
rust_is_dead 4y ago
[dead]
cies 4y ago
I think multi-lingual stemming is the point where I see this as a real ES competitor. Still they've come a long way, and burning too much RAM on ES is not the way fwd either.
- drifteaur 4y ago
  I've had a great experience with Meilisearch, it was very easy to set up. But I'm not sure what's behind the claim that "it supports all languages", aside from handling unicode? Does it support stemming at all? Does it have customized stop words per language?
  - qdequelen 4y ago
    To answer your question precisely, we handle all the space-separated languages and have specific tokenizers for Chinese, Japanese, Korean, Thai, and Hebrew. We plan to add more languages in the future.
dawnerd 4y ago
Very early adopter of meilisearch and it’s pretty great. But bumpy as the team found their footing but overall very impressed with it.
- qdequelen 4y ago
  Thanks for your feedback!
garbagecoder 4y ago
What language you write a program in is not a feature, definitely not a headline one.
- timeon 4y ago
  Maybe not for you. But for my RSS filter it definitely is.
- groestl 4y ago
  Well, choice of language carries _a lot_ of implicit information for which you'd need many more words.
- pkolaczk 4y ago
  Languages are not tools. Languages are materials. If you buy a house you are quite likely interested in what materials were used to build it. There are different features you'd expect from a wooden house vs concrete house.
rsstack 4y ago
Is there a way to run it in WASM, to get something like Lunr[1]? We prefer to do our (small-index, <2MB) search client-side for a bunch of reasons, currently using Lunr.js, but it's a bit annoying and the typeahead search is something I improvised and not really official. [1] https://lunrjs.com/ https://lunrjs.com/
- sandstrom 4y ago
  You could have a look at https://github.com/lucaong/minisearch/ https://github.com/lucaong/minisearch/
  - rsstack 4y ago
    Wow, this might fit our needs much better! Thanks!
    - tmikaeld 4y ago
      Hot tip, we experimented with running minisearch in RAM on cloudflare workers and it works excellent for up to 5MB of index due to it being under the 50ms CPU time. This means, 10M search requests for 5$. The only drawback is that it's expensive to re-index, but if your use case don't require that, it's hard to beat!
    - nickreese 4y ago
      Can vouch for minisearch. Amazing for relative small data that fits in memory. The typeahead is great.
mmachatschek 4y ago
This is awesome news! We've been using meilisearch in production for a few months now and we're more than happy with its reliability. Their work of the last few months really paid off, as the search speed and especially the indexing speed has increased a lot thanks to their efforts. I'm excited to see all the things they'll build in the future.
- tpayet 4y ago
  Thank you Markus <3
sandstrom 4y ago
Great news! Been following along for a while and it's a great project. ElasticSearch needs some competition. For us, there are two things missing for us before we could make the switch: 1. Multi-index search; Standard use-case is searching across e.g. users and companies. Common in many SaaS-applications, where you want a single search field with type-ahead for e.g. contacts/organisations/tasks/events. 2. Decay functions; Basically to gradually phase out results for things based on age, distance or something similar. ElasticSearch has pretty good support for these. https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html#function-decay https://www.elastic.co/guide/en/elasticsearch/reference/curr...
- ferdi05 4y ago
  Thanks for your feedback! The Multi-index search is planned, coded, and will be integrated on v1.1 (scheduled for April). The decay function is really interesting, the team will reach you back to know more about this need :)
- joking 4y ago
  Elastic already has a great competitor called solr, which I prefer on multiple aspects over elastic by the way.
ckok 4y ago
Does it have any kind of master/slave or replication abilities? Couldn't find anything in the docs.
- tpayet 4y ago
  Hello! No we don't yet, we are considering it though.
wiradikusuma 4y ago
I see comparison against other search engines, but how does it compare to RDBMS full text search e.g. Postgre's? I know it's not apple-to-apple, but most people start with RDMBS.
- bayesian_horse 4y ago
  As far as I understand it a search engine like this is meant to perform well on "Human" queries that are hard to formalize. SQL queries, asking for records based on something like field a has to contain b or something like that are easy to formalize and fulfill by an RDBMS. But the SQL queries get hairier and hairier when the query involves multiple fields or even multiple unrelated tables. Or free form text. And those queries are harder to index. On top of all of that, Humans often want things sorted in an order that isn't straight-forward to express in SQL. What is "relevancy"? All of that can be done in SQL, but it's not what RDBMS engines shine at.
schappim 4y ago
We’ve used Meilisearch in production and it is the closest thing to self hosted Algolia you can get, which in itself is pretty amazing. Unfortunately the performance of indexing (constantly changing records) wasn’t great and Meilisearch would fall behind on indexing records for hours. Meilisearch has been amazingly great for projects where records don’t change all that much (eg docs, or even a customer database), but if you have for example a fast paced ecommerce system with 50k records constantly changing (eg product inventory), it falls over pretty quick. We had to transition over to Elastic for this aspect of our app. The other issue we faced is their Rails gems falling out of step with the server, and when fixes came out, the Rails gem was incompatible for a while. I really really hope 1.0 increases performance to the point where it becomes production ready, because the initial out of the box performance (before getting bogged down with indexing) was pretty amazing. Better than Elastic and on par with Algolia. I recommend keeping Meilisearch on your radar. It is going to be great. I wish the best for the Meili team and hope they succeed!
- Kerollmops 4y ago
  Thank you very much for this amazing feedback, really appreciated. We did a lot of improvement to the indexing part of the engine and now can auto-batch updates which gaves incredible improvements. We will continue to work on this in 2023. Can I know the version you were using?
  - nezirus 4y ago
    My experience with indexing is similar. Up to lets say 1M docs it works fine, but after that it goes south. Even with auto-batch I had to manually prepare large bulk updates and wait for completion during inserts to not overload MS. (I am using Rust client). Other than that, it is simply great. Ranking stuff is great, simple, I only need custom weights there, some additional functions (not just asc/desc) and it would be perfect.
    - nacs 4y ago
      I had the same experience. Pro: Meilisearch search speed and memory use was great compared to others (at the cost of large storage requirements but that's the cheapest thing to upgrade). Con: Indexing documents (even with recommended batch sizes) was extremely intensive on the system as the document count increased (upwards of 20 million docs to index). I had to modify the indexing script to completely pause indexing when system load average went too high to prevent the whole server crashing. Also, this 1.0 upgrade apparently requires a full export and import of data if you're upgrading from the previous release? I hope this isn't the case for >= v1.0 releases because I'm not looking forward to exporting/reimporting 200+GB of Meilisearch data files over and over again.
deleted 4y ago
[deleted]
zX41ZdbW 4y ago
You can query Meilisearch directly from ClickHouse with the integrated table function: https://github.com/ClickHouse/ClickHouse/pull/33332 https://github.com/ClickHouse/ClickHouse/pull/33332 This feature was a student project, and I'm not sure if it will find its usage. If you are using Meilisearch with ClickHouse, or if you think this feature is worth something, please let me know.
jvans 4y ago
This looks very cool, nice work. Any plans to support ANN vector searches in the near future?
- qdequelen 4y ago
  Yes, it's planned!
scop 4y ago
Congrats! Question for the team as I see a possible discrepancy on the website. The "Comparisons" page says there is no limit for number of indices (https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html#limits https://docs.meilisearch.com/learn/what_is_meilisearch/compa...) However, the "Limitations" page says there is a limit of ~180 indices (https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html#limits https://docs.meilisearch.com/learn/what_is_meilisearch/compa...) Can you clarify what, if any, are the limitations of # indices?
- ferdi05 4y ago
  Thanks! Indeed we now have a limit, but this limit depends on the OS you use. The limit is 200 on Linux. We found a way to remove this limit in the next version of Meilisearch (v1.1), which will be released in approximately two months. I would like to know the use case for needing more than 200 indexes. We have handled multi-tenant with a single index and multi-tenant tokens. https://docs.meilisearch.com/learn/security/tenant_tokens.html#multitenancy-and-tenant-tokens https://docs.meilisearch.com/learn/security/tenant_tokens.ht...
  - scop 4y ago
    Multi-tenancy is indeed the use case. Our current solution involves keeping each customer's data in a separate index. I'll review the link. Thanks!
heybrendan 4y ago
How would one begin to use this when data is stored in MySQL, MariaDB, or PostgreSQL?
muhammadusman 4y ago
how does this compare to Typesense? I'd like to see which one uses fewer resources for similar performance
- qdequelen 4y ago
  Hey muhammadusman, I'm the Meilisearch's CEO. We have a complete comparison table. Note that it represents our point of view. https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html#comparison-to-alternatives https://docs.meilisearch.com/learn/what_is_meilisearch/compa... Both Meilisearch and Typesense are really different regarding resource consumption and performance. I would say that where Typesense would have a better indexing performance (Meilisearch has recently improved indexation speed), Meilisearch will guarantee a much faster search performance while keeping impressive relevancy. Regarding the consumption, as Typesense is entirely on RAM and Meilisearch is using memory mapping, Meilisearch would take more disk space but less RAM.
networked 4y ago
The most specific criticism I have read of Meilisearch is https://news.ycombinator.com/item?id=32940683 https://news.ycombinator.com/item?id=32940683. It has four points: (1) words beyond 65535 are silently ignored (this is documented in https://docs.meilisearch.com/learn/advanced/known_limitations.html#maximum-number-of-words-per-attribute https://docs.meilisearch.com/learn/advanced/known_limitation... ); (2) the position of a matching word in a document non-optionally affects ranking; (3) to get the match information you must retrieve the entire attribute; (4) the meaning of PUT and POST is switched relative to RFC 7231. Are points (2) through (4) true? Has any of the points been an issue for you in practice?
- Kerollmops 4y ago
  What’s funny is that (1) doesn’t look like a real limit when you know that the first Harry Potter book is nearly 77000 words. The recommended way is to split your documents by paragraph to increase relevancy, this way you can see the exact part that match. About (2) we will work on exposing two new ranking rules to be able to control that. For (3) I thought it was fixed. We decided to implement (4) the PUT and POST this way after looking how others were doing that.
  - networked 4y ago
    Thanks for your reply. I agree about (1). I have checked the datasets I have set up search for, and they either have no or under 1% of documents with more than 65535 words. (This is without any processing to break up the documents into sections.)
slig 4y ago
Is there a way to somehow find related documents to a specific document?
amateurdev0_07 4y ago
Thank you for making MeiliSearch. I use it for a personal project that gets a few hits a day, mostly from me and my friends. https://pulpflakes.com/fmisearch/ https://pulpflakes.com/fmisearch/ It's a search over an index of fiction in the English language, first published in periodicals. Searchable by author, artist, magazine name and specific issue. Biggest index has about 200K documents, doc sizes are tiny. Integrated with my WordPress site by handwritten PHP. Which was fun. Performance is great. I didn't run into too many issues, and those I did i could resolve. What i remember: 1. The rules for text searches are too strict by default and if the order of words is different, will result in no matches. A, B will not return a result if B A is in the database. 2. Creating an index, uploading documents and changing settings required quite a bit of work. A week's worth of coding, almost. Would have loved to have a reasonably robust shell script that could take a JSON file with metadata on index and do the grunt work. 3. I have multiple types of documents, would have liked search to cover all of them so I don't have to change search type manually each time. 4. The default number of documents and max uploaded file size is too low. 200K and 200 MB or something. But it fails even on smaller file size. The above sound like complaints. They're problems I ran into and others might. I love how productive Meilisearch made me. Thank you.
hnaccountme 4y ago
Anyone else having deja vu of when Java did this sort of 'X' build with Java?
survirtual 4y ago
This looks like an effective piece for a project I have. It would be significantly more effective if it was published on crates.io and could be instantiated within Rust, and was able to operate in memory (or have a filesystem passed to it, so that can be simulated) I found this issue which tracks crates.io publication: https://github.com/meilisearch/meilisearch/issues/3367 https://github.com/meilisearch/meilisearch/issues/3367 Would be nice to see that made a priority. Having a powerful search engine that can be embedded in a larger application and made portable (like being able to deploy to WASM) would be extremely novel and valuable. Given Rust is already in use, I think it may not necessarily demand too much effort. When search becomes a focus for what I’m working on, perhaps I will make that happen if not already done yet. Thanks for making this available to people.

448 points by tpayet 4y ago

I tried RTFM'ing, but can Meilisearch handle restricted documents (P&C) and integrated with LDAP/AD to pull security groups? P.S. great to see your documentation search is powered by your own product (!)

fyzix 4y ago

They need to structure their pricing page better. A quick glance had me thinking that $1200 was the minimum for production use. But the free tier is actually pay as you go.

nop_slide 4y ago

This looks really cool and I might try the self hosted option out on my small website as an upgrade from Postgres’ full text search. I was hoping the cloud version would be more appealing, granted there seems to be a generous free tier but the next option is $1200 a month?!

koblas 4y ago
Was excited to see a non-GC'ed search engine that looked solid. But, without having the replicated - distributed version of it in the "free" tier makes it hard to really evaluate.
- tpayet 4y ago
  Feel free to reach out to quentin@meilisearch.com, we'll find a way for you to evaluate the pro plan!
tpayet 4y ago
Sorry, it might not be obvious, but you can go over the free tier and pay for the usage at 0.25$ for each 1000 searches/documents :)
- nop_slide 4y ago
  Ah yep sorry I missed that! Good to know. I just saw the next option was $1200 and my eyes became fixated on the number. Maybe I will try out the cloud version then even though I expect my site would probably be well in the free tier limit, like I said it seems like a very generous tier.
ren_engineer 4y ago
their free tier looks like it has a "pay as you go" option once you exceed it that's identical to the paid option per 1K searches and 1K documents. You are basically paying for priority support, pretty common strategy and seems fair to me. just noticed you don't get high availability on free tier which sucks, but I guess if search is mission critical to the point you need it, you would be willing to pay. Most of these database type companies start off targeting enterprise and then roll out self-serve solutions as they scale.

kacy 4y ago

We’ve been using a Meilisearch for the last six months or so and have been delighted with its performance and usability. It uses a fraction of the resources as Elasticsearch, and the language support is extensive and very active. That being said, our cluster is much smaller than other ones I’ve worked with in the past, so I can’t comment on its reliability at massive scale. I’ve also been very impressed with how active contributors are on GitHub and in their Discord. Everyone seems like good people, and it’s a project I’m excited to keep using.

tpayet 4y ago
Thank you very much! I'll share your comment with the team <3
tempest_ 4y ago
This is the thing I find when people post "ElasticSearch Alternative". 80% of ElasticSearch's value add (wrt search anyway) is all the clustering and frame work that allows you to span the search over tens or hundreds of machines "easily". I think the same is true here. Probably the comparison should be with the underlying search libraries that ES sits on. I suppose this comparison makes sense in a world where most people don't run their own servers much any more since the clustering etc would be a problem for the cloud offering and not the consumer.
- Semaphor 4y ago
  > 80% of ElasticSearch's value add (wrt search anyway) is all the clustering Or configurability. I looked at this again now that 1.0 is out, but besides the .NET client still being in an alpha state, it’s also very zero-configuration. There seems to be no configurability regarding tokenization strategies, for example. Now, I certainly see the appeal, I barely understand my own ES code and meilisearch replicates probably 70% of it with no configuration at all, that’s impressive, but it also means that switching would mean giving up on those 30%.
- kmac_ 4y ago
  Yeah, Elastic also brings advanced aggregates and filters, Kibana, nice UI where you can explore data and create dashboards easily and tons of bigger and smaller features. But in some areas both products are comparable.

manigandham 4y ago

Congrats to the team, it's been interesting to watch the development of Meilisearch (and it's close competitor Typesense). Algolia has really paved the way here but it's nice to see the open-source options with more configurations and better default UX. There's also many search libraries if you want to embed search more deeply into your app. I have a list of modern search systems and libraries here: https://manigandham.com/post/search-systems-libraries https://manigandham.com/post/search-systems-libraries

naiv 4y ago
I will never understand who the target group of Algolia is besides a website where the number of records coincidentally is in the range of the number of queries. At least they got rid of the pricing per indexing transaction which made it even more absurd. If Algolia would offer an instance based pricing on cpu, ram and storage they would be the clear winner imho.
- manigandham 4y ago
  Why do the number of records and searches have to be similar? The current pricing is simple - you pay per "search unit" which scales in both dimensions. The vast majority of small/medium customers would rather pay-as-you-go than maintain a fixed cost instance, and it allows Algolia to efficiently pack them into a multitenant architecture instead of wasting resource overhead.
  - naiv 4y ago
    If you eg index geonames, you have 4 mio. records but you might only have 50.000 queries a month. you pay $4,000 for minimal compute resources, 4GB of RAM and 3 gigabytes of storage space. Would be less but algolia requires you to create a replica for each sort option separately. With 4 mio. records and 4 mio. queries I would pay the same. But then at least have 4 mio. queries. The other way around, if we would just index all 200+ countries in the world and have autocomplete with a lot of visitors we would pay for eg 50.000 users per day typing in 3 letters again $4.000. Same for us, we offer 350.000 movies with 2 mio. scenes. With Typesense or even Elasticsearch Cloud we would pay 5% of what we would pay Algolia.

chimen 4y ago

Is Rust that important that you have to place "built in Rust" in the title? Is this like a cult following that we only bet on traffic and interest coming from other evangelists where Rust is the only feature that matter? 4 months ago: " Meilisearch, open-source alternative to Algolia in Rust lands a $15M Series A" It's not the first time I see, there are at least 2-3 daily submissions reaching the FP in this manner so I'm curious: "built in Rust" = marketing these days?

adamnemecek 4y ago
It is important. Rust projects are infinitely easier to contribute to.
- dividedbyzero 4y ago
  Compared to what?
  - freilanzer 4y ago
    Python?
  - bsnnkv 4y ago
    For me: compared to projects in languages with less mature tooling and compilers less capable of preventing entire classes of errors by default.
  - adamnemecek 4y ago
    Anything. Legit no language comes even close in terms of how easy it is to git clone something and get it to build.
    - hu3 4y ago
      I'd argue Go projects tend to be easier to build since they require nightly Go builds much less frequently (I don't even remember a project that ever required nightly Go tbh). https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=rust%20nightly&sort=byPopularity&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

msvan 4y ago

How does Meilisearch compare to ElasticSearch from an operational point of view? I've experienced ElasticSearch to be quite painful to maintain, requiring lots of manual tweaking to balance shards and careful design of indices.

tpayet 4y ago
That's the point! We don't ambition to compete with Elastic on everything (logs, analytics, etc). We are doing search for front-end users with a strong focus on relevancy, speed & developer experience. You can read a bit more on our documentation https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html#meilisearch-vs-elasticsearch https://docs.meilisearch.com/learn/what_is_meilisearch/compa...
- sidmitra 4y ago
  A quick question, are there any limits around number of separate indexes we can have with meilisearch? I'm thinking atleast say 20-30K separate indexes to start with. My use case is that i want to start creating some indexes that are "per-user" and some "per-company" where a company(customer) might have many users. This is to do some sort of double tenant isolation. I will create different keys that have permission to specific indexes and deliver those to the user somehow. My current solution does hacky things with Elasticsearch like adding query filters by user/company-id attributes in the background automatically. But since meilisearch would be customer facing, i need stronger guarantees around permissions per index. I tried this out a year ago on Meilsearch locally, but haven't stress tested it by creating thousands of them like production. Or is there a better way to do this. This is also a reason where memory-only systems like Typesense didn't make sense to me. I'm fine with taking a performance hit by going to disk to pull the right index. Not every index will be used all the time. I might also look at sharding/partitioning features if present.
  - dureuill 4y ago
    Hello! > A quick question, are there any limits around number of separate indexes we can have with meilisearch? Yes! In v1.0, about 180 indexes under Linux in the same instance[1]. The good news is that I'm personally working on lifting this limitation for v1.1 (planned to release in the beginning of April), which should be able to accommodate an unlimited number of indexes[2] (disk space permits, of course). Note that having many indexes does have an impact on performance and will keep doing so even after v1.1. > Or is there a better way to do this. If it works for your use case, you can try using a single index (or a few indexes) with tenant tokens[3] for multitenancy. Hope this helps :-) [1]: https://docs.meilisearch.com/learn/advanced/known_limitations.html#maximum-database-size https://docs.meilisearch.com/learn/advanced/known_limitation... [2]: https://github.com/meilisearch/meilisearch/issues/3382 https://github.com/meilisearch/meilisearch/issues/3382 [3]: https://docs.meilisearch.com/learn/security/tenant_tokens.html https://docs.meilisearch.com/learn/security/tenant_tokens.ht...

drcongo 4y ago

Congrats team. Meilisearch is an absolute joy to work with.

tpayet 4y ago
Thanks :D
nynapalm 4y ago
Thanks! :)
ferdi05 4y ago
Thanks!

leeoniya 4y ago

compared to https://typesense.org/ https://typesense.org/ ?

bduffany 4y ago
typesense did their own comparison here: https://typesense.org/typesense-vs-algolia-vs-elasticsearch-vs-meilisearch/ https://typesense.org/typesense-vs-algolia-vs-elasticsearch-...
- deleted 4y ago
  [deleted]
- curquiza 4y ago
  Unfortunately, the comparison with Meilisearch is not up to date in this link. Also, we have to keep in mind that every comparison written by a company is always oriented.
  - jabo 4y ago
    I maintain that comparison page on the Typesense side. I just updated it as recently as yesterday, based on my observation. But let me know which ones need updating for Meilisearch. Happy to update. While we’re on the topic, reminder about some of the outdated information in your comparison pages: https://twitter.com/typesense/status/1620825236055932928?s=46&t=ryPI1ZnQYEvrXgPmPl9I5Q https://twitter.com/typesense/status/1620825236055932928?s=4...
    - traverseda 4y ago
      I'd say that the bit where typesense can only work with data that fits in ram is actually a pretty big problem for a lot of use cases, as an aside. That feature alone would discount typesense for basically all of my personal projects. Might be a trade off I'd be willing to make on a professional project given the other features but it seems really wasteful. Personally I find the meilisearch comparison to be more useful for the type of stuff I'm doing: https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html https://docs.meilisearch.com/learn/what_is_meilisearch/compa... Of course I'm not a large enterprise e-commerce site. I'm doing personal projects like web archiving, (dataset probably won't be anywhere near fitting in ram) or I'm using search engines on embedded devices (search needs to play well with others, not use all my ram).

tmikaeld 4y ago

My team tried to use Meilisearch for large datasets, unfortunately, it's impossible to plan the RAM usage. If you have very little searches, it consumed very little, but if you have a lot of search traffic, it may consume more than we could provision beforehand. This made it too unpredictable and too expensive, so we went with Manticore instead. I don't know if this has been addressed in 1.0, hopefully it has.

marban 4y ago
Do you have any numeric definitions for few and lots?
- traverseda 4y ago
  I think that they might have fixed it. I noted this as a problem with earlier meilisearch releases as well, but reading through the documentation it looks like they don't require the entire index to be in memory any more, allowing it to be a memory mapped file. https://docs.meilisearch.com/learn/advanced/storage.html#lmdb https://docs.meilisearch.com/learn/advanced/storage.html#lmd... >For the best performance, it is recommended to provide the same amount of RAM as the size the database takes on disk, so all the data structures can fit in memory. > [...] >It is important to note that there is no reliable way to predict the final size of a database. This is true for just about any search engine on the market—we're just the only ones saying it out loud. Looks like a 10MB document is taking ~200MB, from their docs. I don't think that scales linearly though, since it's a reverse index it is going to scale based on the number of unique words it finds, with each document adding a bit on top of that. You'd expect it to have a pretty big index to cover common english words, and then each document adds a bit on top of that. Definitely seems like somewhere they could make some improvements though. Some transparent compression could probably help, and with zstd's dictionary feature it can be fine tuned to the data they're actually seeing. Not about to replace xapian in kiwix (offline wikipedia reader) any time soon, I think.
- tmikaeld 4y ago
  Our index was aimed at handling 20 000 documents at total of 35MB of CSV, this would balloon into 0.7GB to 1GB of RAM and we expected at least 1000 of these indexes, which would require dedicated servers with 1TB of RAM. This was when Meili was at version 0.27. With manticore, we've tried to run into these issues in benchmarks, but the only problem we got was temporary high IO load when indexes need to be re-indexed with new or changed documents. In total it's at 50-70% of the RAM usage compared to Meili. We'd be happy to re-visit, but looking at the docs - it seems to be about the same as it was back then (a year ago).

kristiandupont 4y ago

I am using the core (called "Milli") in a local indexer that I run on my repositories and Obsidian files. It works like a charm and I am very happy with it. Obviously that's a use case with very little traffic but just indexing my repositories folder is quite a bit of work and it does it surprisingly fast. The only real thing I am missing is a typeahead feature.

dureuill 4y ago
Hello from a Meilisearch team member, wow your project looks very interesting. How do you handle things like the filesystem changing while your indexer is offline? Do you reindex from scratch at startup? Regarding typeahead, is this what we call "query suggestions"[1]? At the moment, we think that this is something that frontends and SDK can provide rather than the engine, so that means you wouldn't find it at the Milli level. We think you could maybe build an ancillary suggestion index and make two queries instead of one when typing, so as to get both results and suggestions at once. Here's a chat link[2] to our latest discussions on the topic; feel free to come and weigh in if you're interested! [1]: https://roadmap.meilisearch.com/c/31-query-suggestions https://roadmap.meilisearch.com/c/31-query-suggestions [2]: https://discord.com/channels/1006923006964154428/1068507365801992272/1068507369895633056 https://discord.com/channels/1006923006964154428/10685073658...
- kristiandupont 4y ago
  Thank you! Yes, I reindex. I store the file timestamp along with the contents, so it's not quite as involved as it could seem but startup does take a bit. And, I don't have a good way of discovering deleted files at the moment. Not a big deal as it is, but something I will look into. And yes, query suggestions are exactly what I mean. Thank you for informing me, I guess I will have to look into how I can make it myself :-)
  - dureuill 4y ago
    You could maybe use something equivalent to the "index hot swap"[1] feature we have at the Meilisearch level at startup, so that you make the reindexing in a another index at startup, and then atomatically swap this fresh index with the old one when it is ready? That way, you have fast startup at the cost of having possibly out-of-date information for a while after startup. (you could even reindex from scratch completely in the background at startup, so no need to discover deleted files at all) [1]: https://blog.meilisearch.com/zero-downtime-index-deployment/ https://blog.meilisearch.com/zero-downtime-index-deployment/

snowpid 4y ago

Can you rewrite it in Rust?

curquiza 4y ago
We will think about it :D

rust_is_dead 4y ago

[dead]

cies 4y ago

I think multi-lingual stemming is the point where I see this as a real ES competitor. Still they've come a long way, and burning too much RAM on ES is not the way fwd either.

drifteaur 4y ago
I've had a great experience with Meilisearch, it was very easy to set up. But I'm not sure what's behind the claim that "it supports all languages", aside from handling unicode? Does it support stemming at all? Does it have customized stop words per language?
- qdequelen 4y ago
  To answer your question precisely, we handle all the space-separated languages and have specific tokenizers for Chinese, Japanese, Korean, Thai, and Hebrew. We plan to add more languages in the future.

dawnerd 4y ago

Very early adopter of meilisearch and it’s pretty great. But bumpy as the team found their footing but overall very impressed with it.

qdequelen 4y ago
Thanks for your feedback!

garbagecoder 4y ago

What language you write a program in is not a feature, definitely not a headline one.

timeon 4y ago
Maybe not for you. But for my RSS filter it definitely is.
groestl 4y ago
Well, choice of language carries _a lot_ of implicit information for which you'd need many more words.
pkolaczk 4y ago
Languages are not tools. Languages are materials. If you buy a house you are quite likely interested in what materials were used to build it. There are different features you'd expect from a wooden house vs concrete house.

rsstack 4y ago

Is there a way to run it in WASM, to get something like Lunr[1]? We prefer to do our (small-index, <2MB) search client-side for a bunch of reasons, currently using Lunr.js, but it's a bit annoying and the typeahead search is something I improvised and not really official. [1] https://lunrjs.com/ https://lunrjs.com/

sandstrom 4y ago
You could have a look at https://github.com/lucaong/minisearch/ https://github.com/lucaong/minisearch/
- rsstack 4y ago
  Wow, this might fit our needs much better! Thanks!
  - tmikaeld 4y ago
    Hot tip, we experimented with running minisearch in RAM on cloudflare workers and it works excellent for up to 5MB of index due to it being under the 50ms CPU time. This means, 10M search requests for 5$. The only drawback is that it's expensive to re-index, but if your use case don't require that, it's hard to beat!
  - nickreese 4y ago
    Can vouch for minisearch. Amazing for relative small data that fits in memory. The typeahead is great.

mmachatschek 4y ago

This is awesome news! We've been using meilisearch in production for a few months now and we're more than happy with its reliability. Their work of the last few months really paid off, as the search speed and especially the indexing speed has increased a lot thanks to their efforts. I'm excited to see all the things they'll build in the future.

tpayet 4y ago
Thank you Markus <3

sandstrom 4y ago

Great news! Been following along for a while and it's a great project. ElasticSearch needs some competition. For us, there are two things missing for us before we could make the switch: 1. Multi-index search; Standard use-case is searching across e.g. users and companies. Common in many SaaS-applications, where you want a single search field with type-ahead for e.g. contacts/organisations/tasks/events. 2. Decay functions; Basically to gradually phase out results for things based on age, distance or something similar. ElasticSearch has pretty good support for these. https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html#function-decay https://www.elastic.co/guide/en/elasticsearch/reference/curr...

ferdi05 4y ago
Thanks for your feedback! The Multi-index search is planned, coded, and will be integrated on v1.1 (scheduled for April). The decay function is really interesting, the team will reach you back to know more about this need :)
joking 4y ago
Elastic already has a great competitor called solr, which I prefer on multiple aspects over elastic by the way.

ckok 4y ago

Does it have any kind of master/slave or replication abilities? Couldn't find anything in the docs.

tpayet 4y ago
Hello! No we don't yet, we are considering it though.

wiradikusuma 4y ago

I see comparison against other search engines, but how does it compare to RDBMS full text search e.g. Postgre's? I know it's not apple-to-apple, but most people start with RDMBS.

bayesian_horse 4y ago
As far as I understand it a search engine like this is meant to perform well on "Human" queries that are hard to formalize. SQL queries, asking for records based on something like field a has to contain b or something like that are easy to formalize and fulfill by an RDBMS. But the SQL queries get hairier and hairier when the query involves multiple fields or even multiple unrelated tables. Or free form text. And those queries are harder to index. On top of all of that, Humans often want things sorted in an order that isn't straight-forward to express in SQL. What is "relevancy"? All of that can be done in SQL, but it's not what RDBMS engines shine at.

schappim 4y ago

We’ve used Meilisearch in production and it is the closest thing to self hosted Algolia you can get, which in itself is pretty amazing. Unfortunately the performance of indexing (constantly changing records) wasn’t great and Meilisearch would fall behind on indexing records for hours. Meilisearch has been amazingly great for projects where records don’t change all that much (eg docs, or even a customer database), but if you have for example a fast paced ecommerce system with 50k records constantly changing (eg product inventory), it falls over pretty quick. We had to transition over to Elastic for this aspect of our app. The other issue we faced is their Rails gems falling out of step with the server, and when fixes came out, the Rails gem was incompatible for a while. I really really hope 1.0 increases performance to the point where it becomes production ready, because the initial out of the box performance (before getting bogged down with indexing) was pretty amazing. Better than Elastic and on par with Algolia. I recommend keeping Meilisearch on your radar. It is going to be great. I wish the best for the Meili team and hope they succeed!

Kerollmops 4y ago
Thank you very much for this amazing feedback, really appreciated. We did a lot of improvement to the indexing part of the engine and now can auto-batch updates which gaves incredible improvements. We will continue to work on this in 2023. Can I know the version you were using?
- nezirus 4y ago
  My experience with indexing is similar. Up to lets say 1M docs it works fine, but after that it goes south. Even with auto-batch I had to manually prepare large bulk updates and wait for completion during inserts to not overload MS. (I am using Rust client). Other than that, it is simply great. Ranking stuff is great, simple, I only need custom weights there, some additional functions (not just asc/desc) and it would be perfect.
  - nacs 4y ago
    I had the same experience. Pro: Meilisearch search speed and memory use was great compared to others (at the cost of large storage requirements but that's the cheapest thing to upgrade). Con: Indexing documents (even with recommended batch sizes) was extremely intensive on the system as the document count increased (upwards of 20 million docs to index). I had to modify the indexing script to completely pause indexing when system load average went too high to prevent the whole server crashing. Also, this 1.0 upgrade apparently requires a full export and import of data if you're upgrading from the previous release? I hope this isn't the case for >= v1.0 releases because I'm not looking forward to exporting/reimporting 200+GB of Meilisearch data files over and over again.

deleted 4y ago

[deleted]

zX41ZdbW 4y ago

You can query Meilisearch directly from ClickHouse with the integrated table function: https://github.com/ClickHouse/ClickHouse/pull/33332 https://github.com/ClickHouse/ClickHouse/pull/33332 This feature was a student project, and I'm not sure if it will find its usage. If you are using Meilisearch with ClickHouse, or if you think this feature is worth something, please let me know.

jvans 4y ago

This looks very cool, nice work. Any plans to support ANN vector searches in the near future?

qdequelen 4y ago
Yes, it's planned!

scop 4y ago

Congrats! Question for the team as I see a possible discrepancy on the website. The "Comparisons" page says there is no limit for number of indices (https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html#limits https://docs.meilisearch.com/learn/what_is_meilisearch/compa...) However, the "Limitations" page says there is a limit of ~180 indices (https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html#limits https://docs.meilisearch.com/learn/what_is_meilisearch/compa...) Can you clarify what, if any, are the limitations of # indices?

ferdi05 4y ago
Thanks! Indeed we now have a limit, but this limit depends on the OS you use. The limit is 200 on Linux. We found a way to remove this limit in the next version of Meilisearch (v1.1), which will be released in approximately two months. I would like to know the use case for needing more than 200 indexes. We have handled multi-tenant with a single index and multi-tenant tokens. https://docs.meilisearch.com/learn/security/tenant_tokens.html#multitenancy-and-tenant-tokens https://docs.meilisearch.com/learn/security/tenant_tokens.ht...
- scop 4y ago
  Multi-tenancy is indeed the use case. Our current solution involves keeping each customer's data in a separate index. I'll review the link. Thanks!

heybrendan 4y ago

How would one begin to use this when data is stored in MySQL, MariaDB, or PostgreSQL?

muhammadusman 4y ago

how does this compare to Typesense? I'd like to see which one uses fewer resources for similar performance

qdequelen 4y ago
Hey muhammadusman, I'm the Meilisearch's CEO. We have a complete comparison table. Note that it represents our point of view. https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html#comparison-to-alternatives https://docs.meilisearch.com/learn/what_is_meilisearch/compa... Both Meilisearch and Typesense are really different regarding resource consumption and performance. I would say that where Typesense would have a better indexing performance (Meilisearch has recently improved indexation speed), Meilisearch will guarantee a much faster search performance while keeping impressive relevancy. Regarding the consumption, as Typesense is entirely on RAM and Meilisearch is using memory mapping, Meilisearch would take more disk space but less RAM.

networked 4y ago

The most specific criticism I have read of Meilisearch is https://news.ycombinator.com/item?id=32940683 https://news.ycombinator.com/item?id=32940683. It has four points: (1) words beyond 65535 are silently ignored (this is documented in https://docs.meilisearch.com/learn/advanced/known_limitations.html#maximum-number-of-words-per-attribute https://docs.meilisearch.com/learn/advanced/known_limitation... ); (2) the position of a matching word in a document non-optionally affects ranking; (3) to get the match information you must retrieve the entire attribute; (4) the meaning of PUT and POST is switched relative to RFC 7231. Are points (2) through (4) true? Has any of the points been an issue for you in practice?

Kerollmops 4y ago
What’s funny is that (1) doesn’t look like a real limit when you know that the first Harry Potter book is nearly 77000 words. The recommended way is to split your documents by paragraph to increase relevancy, this way you can see the exact part that match. About (2) we will work on exposing two new ranking rules to be able to control that. For (3) I thought it was fixed. We decided to implement (4) the PUT and POST this way after looking how others were doing that.
- networked 4y ago
  Thanks for your reply. I agree about (1). I have checked the datasets I have set up search for, and they either have no or under 1% of documents with more than 65535 words. (This is without any processing to break up the documents into sections.)

slig 4y ago

Is there a way to somehow find related documents to a specific document?

amateurdev0_07 4y ago

Thank you for making MeiliSearch. I use it for a personal project that gets a few hits a day, mostly from me and my friends. https://pulpflakes.com/fmisearch/ https://pulpflakes.com/fmisearch/ It's a search over an index of fiction in the English language, first published in periodicals. Searchable by author, artist, magazine name and specific issue. Biggest index has about 200K documents, doc sizes are tiny. Integrated with my WordPress site by handwritten PHP. Which was fun. Performance is great. I didn't run into too many issues, and those I did i could resolve. What i remember: 1. The rules for text searches are too strict by default and if the order of words is different, will result in no matches. A, B will not return a result if B A is in the database. 2. Creating an index, uploading documents and changing settings required quite a bit of work. A week's worth of coding, almost. Would have loved to have a reasonably robust shell script that could take a JSON file with metadata on index and do the grunt work. 3. I have multiple types of documents, would have liked search to cover all of them so I don't have to change search type manually each time. 4. The default number of documents and max uploaded file size is too low. 200K and 200 MB or something. But it fails even on smaller file size. The above sound like complaints. They're problems I ran into and others might. I love how productive Meilisearch made me. Thank you.

hnaccountme 4y ago

Anyone else having deja vu of when Java did this sort of 'X' build with Java?

survirtual 4y ago

This looks like an effective piece for a project I have. It would be significantly more effective if it was published on crates.io and could be instantiated within Rust, and was able to operate in memory (or have a filesystem passed to it, so that can be simulated) I found this issue which tracks crates.io publication: https://github.com/meilisearch/meilisearch/issues/3367 https://github.com/meilisearch/meilisearch/issues/3367 Would be nice to see that made a priority. Having a powerful search engine that can be embedded in a larger application and made portable (like being able to deploy to WASM) would be extremely novel and valuable. Given Rust is already in use, I think it may not necessarily demand too much effort. When search becomes a focus for what I’m working on, perhaps I will make that happen if not already done yet. Thanks for making this available to people.