3 ms·

Hi everyone — I'm one of the maintainers of Manticore Search. Huge thanks to @klaussilveira for submitting this — we really appreciate the interest and the tho

Hi everyone — I'm one of the maintainers of Manticore Search. Huge thanks to @klaussilveira for submitting this — we really appreciate the interest and the thoughtful discussion here. A few points that came up in the thread and are worth clarifying: - We do get compared to Elasticsearch a lot. While we support some of its APIs, Manticore isn't a drop-in replacement. We've focused on performance, simplicity, and keeping things open-source without vendor lock-in. Our own SQL-based query language and REST endpoints are part of that philosophy. - @mdaniel was right to question the "drop-in" wording — that's not our goal. - As @sandstrom pointed out, tools like Typesense and Meilisearch are part of this evolving search space. We see Manticore fitting in where users want powerful full-text and vector search capabilities with lower resource overhead and SQL capabilities (we support JSON too though) We'd love to hear from you: - What are your main use cases for search or log indexing? - Which Elasticsearch features (if any) are absolutely essential for you? - Are there performance comparisons or scaling challenges you'd like to see addressed? Happy to answer any questions or dive deeper.

aaroninsf 1y ago
Since you asked! I don't see mention of Lucene on the repo landing page, could you ELI5 the query language and TD-IDF? (Being lazy, I am happy to look into this myself lol.)
- snikolaev 1y ago
  We made a blogpost "TF-IDF in a nutshell" - https://manticoresearch.com/blog/tf-idf-in-a-nutshell/ https://manticoresearch.com/blog/tf-idf-in-a-nutshell/ Manticore Search's query language is more expressive than Lucene's. While Lucene supports basic boolean logic, field search, phrase queries, wildcards, and proximity, Manticore adds many powerful operators. These include quorum matching (/N), strict word order (<<), NEAR/NOTNEAR proximity, MAYBE (soft OR), exact form (=word), position anchors (^word, word$), full regular expressions, and more. Manticore uses SQL-style syntax with a MATCH() clause for full-text search, making it easier to combine text search with filters and ranking.
Semaphor 1y ago
> What are your main use cases for search Camera and camera lens names. I tried mellisearch (1-2 years ago), and while I loved the simplicity (I barely understand what I threw together with many, many lines of C# code for elastic search; this is partially on ES, but clearly on me as well), it was not good at getting results. Names like "Lumix DC-S1IIE", "DSC-RX100 VIIA", or "FE 50-150 mm F2 GM (SEL50150GM)" do not quite work with default tokenizers and analyzers. Of course that is for product names, for full text queries still need to use normal language rules… except for product names showing up in the text, so now I need multiple indexes for every field, and searching different sub-fields sometimes with different query analyzers. It was a lot of trial and error getting ES to both find what was searched for, but also be typo tolerant. It’s very easy getting far too many results, and bad scoring for fuzzy queries. So a bit of a special case, but something the customization capabilities of ES support pretty well. Luckily, our dataset is rather small, maybe 100k documents, so scalability is not a problem.
- snikolaev 1y ago
  Thanks for sharing! What sets Manticore apart from Meilisearch and Elasticsearch is that it lets you configure tokenization at a low level by: - choosing which characters should be treated as token characters, and using the rest as token separators - defining "blend chars" — for example, the hyphen (-) could make sense as both a separator and a non-separator in your case - or optionally adding it to the ignore_chars list - there's also regexp_filter to process tokens when indexing and searching That said, setting things like this up perfectly is always tricky with any search engine, because the words and punctuation in real data often don't follow regular patterns. It's especially difficult when you want to find "abc def" by "ab cd ef" which may be a common situation in your case.
sontek 1y ago
The main tag line says "Drop-in replacement for E in the ELK stack" -- but here you say you aren't a drop-in replacement. Does this mean you've at least implemented every API that Kibana requires?
- snikolaev 1y ago
  Not every, but Kibana can be used with Manticore with some limitations - https://github.com/manticoresoftware/kibana-demo https://github.com/manticoresoftware/kibana-demo
  - Keyframe 1y ago
    So, "Drop-in replacement for E in the ELK stack, but not really, maybe, limitations apply"? Why even use that copy then?
    - esafak 1y ago
      To entice people searching for ElasticSearch alternatives.
      - xeraa 1y ago
        The clear misrepresentation for years with absolutely no shame...
    - victorbjorklund 1y ago
      To be honest plenty do that. Companies claim their object storage is S3 compatiable even if they havent implemented every single functionality that S3 has.
    - nine_k 1y ago
      Let's call it an almost equivalent replacement, like "e" for "E".
sandstrom 1y ago
"What are your main use cases for search or log indexing?" To me, storing and searching logs is quite different from most other search use-cases, and it's not obvious that they should be handled by the same piece of software. For example, tokenization, stemming, language support many other things and are basically useless in log search. Also, log search is often storing a lot of data, and rarely retrieving it (different usage pattern from many search use-cases which tend to be less write-heavy and more about reads). I know ElasticSearch has had success in both, but if I were Manticore/Typesense/Meilisearch I'd probably just skip logs altogether. Loki, QuickWit and other such tools are likely better suited for logs. - https://github.com/quickwit-oss/quickwit https://github.com/quickwit-oss/quickwit - https://github.com/grafana/loki https://github.com/grafana/loki
lokl 1y ago
I've been using Sphinx for 20 years for full-text search with a custom stemmer. I considered switching to Manticore, but didn't see a huge need to do so, because Sphinx still works well for me and requires zero maintenance. Any big new features that might entice me to switch? (I only have a few GB of indexes, covering a few million documents.)
- snikolaev 1y ago
  If your setup works fine and doesn't need any maintenance, there's really no reason to switch to something else. But once you upgrade to a newer version of Sphinx and it crashes, I personally feel more comfortable knowing I can report the issue on GitHub and expect it to be fixed eventually. Unfortunately, that's not how it works with Sphinx. Speaking of features, both Sphinx (as a closed-source project) and Manticore (as an open-source project) have added some nice improvements during last years. But again, if you're happy with the 20-year-old version, there's probably nothing to worry about.
  - nchmy 1y ago
    Im a fan of manticore and promote it when I can, but this doesn't seem like an approoriate response. It appears generous on the surface, but has various unnecessary and subtle digs in it - all while it would be easy to show how manticore is a more appropriate choice. They're surely not using a 20-year-old version of Sphinx - it appears that the latest open source version on Github is 8 years old. They have a closed source version which appears to be maintained, though has infrequent releases. Why not just say these things, which make manticore self-evidently a better choice? https://github.com/sphinxsearch/sphinx https://github.com/sphinxsearch/sphinx https://sphinxsearch.com/downloads/current/ https://sphinxsearch.com/downloads/current/
cbsmith 1y ago
> We do get compared to Elasticsearch a lot. While we support some of its APIs, Manticore isn't a drop-in replacement. Thank you for saying that up front. I read a description of your product and the first thing I thought was, "this looks like a potential alternative to ElasticSearch, but it is not a drop-in replacement for ElasticSearch".

aaroninsf 1y ago

Since you asked! I don't see mention of Lucene on the repo landing page, could you ELI5 the query language and TD-IDF? (Being lazy, I am happy to look into this myself lol.)

snikolaev 1y ago
We made a blogpost "TF-IDF in a nutshell" - https://manticoresearch.com/blog/tf-idf-in-a-nutshell/ https://manticoresearch.com/blog/tf-idf-in-a-nutshell/ Manticore Search's query language is more expressive than Lucene's. While Lucene supports basic boolean logic, field search, phrase queries, wildcards, and proximity, Manticore adds many powerful operators. These include quorum matching (/N), strict word order (<<), NEAR/NOTNEAR proximity, MAYBE (soft OR), exact form (=word), position anchors (^word, word$), full regular expressions, and more. Manticore uses SQL-style syntax with a MATCH() clause for full-text search, making it easier to combine text search with filters and ranking.

Semaphor 1y ago

> What are your main use cases for search Camera and camera lens names. I tried mellisearch (1-2 years ago), and while I loved the simplicity (I barely understand what I threw together with many, many lines of C# code for elastic search; this is partially on ES, but clearly on me as well), it was not good at getting results. Names like "Lumix DC-S1IIE", "DSC-RX100 VIIA", or "FE 50-150 mm F2 GM (SEL50150GM)" do not quite work with default tokenizers and analyzers. Of course that is for product names, for full text queries still need to use normal language rules… except for product names showing up in the text, so now I need multiple indexes for every field, and searching different sub-fields sometimes with different query analyzers. It was a lot of trial and error getting ES to both find what was searched for, but also be typo tolerant. It’s very easy getting far too many results, and bad scoring for fuzzy queries. So a bit of a special case, but something the customization capabilities of ES support pretty well. Luckily, our dataset is rather small, maybe 100k documents, so scalability is not a problem.

snikolaev 1y ago
Thanks for sharing! What sets Manticore apart from Meilisearch and Elasticsearch is that it lets you configure tokenization at a low level by: - choosing which characters should be treated as token characters, and using the rest as token separators - defining "blend chars" — for example, the hyphen (-) could make sense as both a separator and a non-separator in your case - or optionally adding it to the ignore_chars list - there's also regexp_filter to process tokens when indexing and searching That said, setting things like this up perfectly is always tricky with any search engine, because the words and punctuation in real data often don't follow regular patterns. It's especially difficult when you want to find "abc def" by "ab cd ef" which may be a common situation in your case.

sontek 1y ago

The main tag line says "Drop-in replacement for E in the ELK stack" -- but here you say you aren't a drop-in replacement. Does this mean you've at least implemented every API that Kibana requires?

snikolaev 1y ago
Not every, but Kibana can be used with Manticore with some limitations - https://github.com/manticoresoftware/kibana-demo https://github.com/manticoresoftware/kibana-demo
- Keyframe 1y ago
  So, "Drop-in replacement for E in the ELK stack, but not really, maybe, limitations apply"? Why even use that copy then?
  - esafak 1y ago
    To entice people searching for ElasticSearch alternatives.
    - xeraa 1y ago
      The clear misrepresentation for years with absolutely no shame...
  - victorbjorklund 1y ago
    To be honest plenty do that. Companies claim their object storage is S3 compatiable even if they havent implemented every single functionality that S3 has.
  - nine_k 1y ago
    Let's call it an almost equivalent replacement, like "e" for "E".

sandstrom 1y ago

"What are your main use cases for search or log indexing?" To me, storing and searching logs is quite different from most other search use-cases, and it's not obvious that they should be handled by the same piece of software. For example, tokenization, stemming, language support many other things and are basically useless in log search. Also, log search is often storing a lot of data, and rarely retrieving it (different usage pattern from many search use-cases which tend to be less write-heavy and more about reads). I know ElasticSearch has had success in both, but if I were Manticore/Typesense/Meilisearch I'd probably just skip logs altogether. Loki, QuickWit and other such tools are likely better suited for logs. - https://github.com/quickwit-oss/quickwit https://github.com/quickwit-oss/quickwit - https://github.com/grafana/loki https://github.com/grafana/loki

lokl 1y ago

I've been using Sphinx for 20 years for full-text search with a custom stemmer. I considered switching to Manticore, but didn't see a huge need to do so, because Sphinx still works well for me and requires zero maintenance. Any big new features that might entice me to switch? (I only have a few GB of indexes, covering a few million documents.)

snikolaev 1y ago
If your setup works fine and doesn't need any maintenance, there's really no reason to switch to something else. But once you upgrade to a newer version of Sphinx and it crashes, I personally feel more comfortable knowing I can report the issue on GitHub and expect it to be fixed eventually. Unfortunately, that's not how it works with Sphinx. Speaking of features, both Sphinx (as a closed-source project) and Manticore (as an open-source project) have added some nice improvements during last years. But again, if you're happy with the 20-year-old version, there's probably nothing to worry about.
- nchmy 1y ago
  Im a fan of manticore and promote it when I can, but this doesn't seem like an approoriate response. It appears generous on the surface, but has various unnecessary and subtle digs in it - all while it would be easy to show how manticore is a more appropriate choice. They're surely not using a 20-year-old version of Sphinx - it appears that the latest open source version on Github is 8 years old. They have a closed source version which appears to be maintained, though has infrequent releases. Why not just say these things, which make manticore self-evidently a better choice? https://github.com/sphinxsearch/sphinx https://github.com/sphinxsearch/sphinx https://sphinxsearch.com/downloads/current/ https://sphinxsearch.com/downloads/current/

cbsmith 1y ago

> We do get compared to Elasticsearch a lot. While we support some of its APIs, Manticore isn't a drop-in replacement. Thank you for saying that up front. I read a description of your product and the first thing I thought was, "this looks like a potential alternative to ElasticSearch, but it is not a drop-in replacement for ElasticSearch".