4 ms·

Wow it's weird that this comes up, I'm actually running a site I am going to repost to HN today that I want to use as a testbed for search engines (kind of like

Wow it's weird that this comes up, I'm actually running a site I am going to repost to HN today that I want to use as a testbed for search engines (kind of like an extension to my recent collaboration with supabase[0]). Right now I've got the site going on just Postgres FTS + trigram and it's pretty darn fast, looks like I need to test sonic too. Going to burn some midnight oil (in my timezone, anyway) and get it out -- though sonic isn't implemented yet! Anyway to make this comment useful to people, here's my short list of engines that I want to run in parallel: - MeiliSearch (https://github.com/meilisearch/MeiliSearch https://github.com/meilisearch/MeiliSearch) - TypeSense (https://github.com/typesense/typesense https://github.com/typesense/typesense) - Lyra (https://github.com/LyraSearch/lyra https://github.com/LyraSearch/lyra) - OpenSearch (https://github.com/opensearch-project/OpenSearch https://github.com/opensearch-project/OpenSearch) - ZincSearch (https://github.com/prabhatsharma/zinc https://github.com/prabhatsharma/zinc) - Sonic (https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic) There isn't enough out there comparing all these for the simple typical fuzzy search/search box usecase, so I'm adapting a little podcast search site I made to try and use all of these at the same time. So far only Postgres though, will try and add Meilisearch today and post it! Like other people are pointing out, most of these engines won't have all the features of ES (or more accurately Lucene) but I am pretty convinced that most of the time it doesn't actually matter and if someone is searching on your site excessively maybe there's a problem with your UX (unless you're a search engine or repository of information). [0]: https://supabase.com/blog/postgres-full-text-search-vs-the-rest https://supabase.com/blog/postgres-full-text-search-vs-the-r...

hawski 4y ago
Thank you for this comparison. I would also like to know how Bleve Search (https://github.com/blevesearch/bleve https://github.com/blevesearch/bleve) turns out. I have for many years now a small search engine project in my free-time pipeline, but I'm before crawling even and I intend to sit for searching part after some of that.
- hardwaresofton 4y ago
  You're right I should put bleve on there as well. This isn't even the whole list. Toshi (https://github.com/toshi-search/Toshi https://github.com/toshi-search/Toshi) is also out there...
  - snikolaev 4y ago
    If you decide to add Manticore Search to the list feel free to ping me at sergey@manticoresearch.com if you need help with preparing the ingestion scripts etc.
    - hardwaresofton 4y ago
      Oh! Damn it I forgot about manticore -- I had seen it before but forgot to include it. Eventually all of these projects will be highlighted on Awesome F/OSS (https://awsmfoss.com https://awsmfoss.com), but for now I'm just going to dump my bookmarks here for other people, since I'm leaving awesome projects out: Search Engines AWS OpenSearch https://github.com/opensearch-project/OpenSearch https://github.com/opensearch-project/OpenSearch https://github.com/opensearch-project/OpenSearch-Dashboards https://github.com/opensearch-project/OpenSearch-Dashboards https://github.com/opensearch-project/perftop https://github.com/opensearch-project/perftop https://github.com/go-ego/riot https://github.com/go-ego/riot https://groonga.org/ https://groonga.org/ https://github.com/groonga/groonga https://github.com/groonga/groonga https://github.com/meilisearch/MeiliSearch https://github.com/meilisearch/MeiliSearch https://github.com/mosuka/bayard https://github.com/mosuka/bayard https://github.com/nezaboodka/nevod https://github.com/nezaboodka/nevod https://github.com/searx/searx https://github.com/searx/searx https://github.com/stryku/okon https://github.com/stryku/okon https://github.com/toshi-search/Toshi https://github.com/toshi-search/Toshi https://github.com/typesense/typesense https://github.com/typesense/typesense https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic Algolia https://github.com/marconi1992/algolite https://github.com/marconi1992/algolite https://quickwit.io/ https://quickwit.io/ https://github.com/quickwit-inc/quickwit https://github.com/quickwit-inc/quickwit https://docs.meilisearch.com/ https://docs.meilisearch.com/ https://github.com/prabhatsharma/zinc https://github.com/prabhatsharma/zinc phalanx https://github.com/blugelabs/bluge https://github.com/blugelabs/bluge https://github.com/mosuka/phalanx https://github.com/mosuka/phalanx https://github.com/mosuka/blast https://github.com/mosuka/blast ManticoreSearch https://github.com/manticoresoftware/manticoresearch https://github.com/manticoresoftware/manticoresearch https://github.com/manticoresoftware/docker https://github.com/manticoresoftware/docker https://manticoresearch.com/blog/manticore-alternative-to-elasticsearch/ https://manticoresearch.com/blog/manticore-alternative-to-el... https://manticoresearch.com/ https://manticoresearch.com/ https://manual.manticoresearch.com/Introduction https://manual.manticoresearch.com/Introduction https://forum.manticoresearch.com/t/manticore-search-cheatsheets/1049 https://forum.manticoresearch.com/t/manticore-search-cheatsh... https://forum.manticoresearch.com/ https://forum.manticoresearch.com/ Whoosh https://whoosh.readthedocs.io/en/latest/ https://whoosh.readthedocs.io/en/latest/ https://pypi.org/project/Whoosh/ https://pypi.org/project/Whoosh/ lyra https://github.com/nearform/lyra https://github.com/nearform/lyra https://nearform.github.io/lyra/ https://nearform.github.io/lyra/ https://github.com/LyraSearch/lyra https://github.com/LyraSearch/lyra https://lyrasearch.io/ https://lyrasearch.io/ flexsearch https://github.com/nextapps-de/flexsearch#performance-benchmark-ranking https://github.com/nextapps-de/flexsearch#performance-benchm... https://pagefind.app/docs/ https://pagefind.app/docs/ Lucene https://github.com/apache/lucene https://github.com/apache/lucene https://lucene.apache.org/ https://lucene.apache.org/ ZincSearch https://zincsearch.com/ https://zincsearch.com/ Solr https://solr.apache.org/ https://solr.apache.org/ https://solr.apache.org/operator/ https://solr.apache.org/operator/ https://solr.apache.org/guide/solr/latest/getting-started/solr-tutorial.html https://solr.apache.org/guide/solr/latest/getting-started/so... https://github.com/apache/solr https://github.com/apache/solr https://solr.apache.org/guide/solr/latest/deployment-guide/solr-in-docker.html https://solr.apache.org/guide/solr/latest/deployment-guide/s... Konnu https://gitlab.com/shadowislord/konnu https://gitlab.com/shadowislord/konnu Quickwit QuickWit + Clickhouse https://clickhouse.com/docs/en/guides/developer/full-text-search/ https://clickhouse.com/docs/en/guides/developer/full-text-se... https://clickhouse.com/docs/en/sql-reference/functions/string-search-functions/ https://clickhouse.com/docs/en/sql-reference/functions/strin... There is no way I can get to running all of these (this project was supposed to be quick!!), but I will run the ones I noted earlier, and probably manticore too since it was high on my list since it's quite polished looking.
Bilal_io 4y ago
Hey that's a great list of tools. Are you aware of any that can be used client side like Lyra and supports faceted search? I've been looking for a solution and cannot find it, even an algorithm and/or a data structure can be helpful. I attempted coming up with a solution myself but ended up with frustration when it came to making the facets dynamic and update as other filters are applied. I read a couple of papers and one stood out [0], which introduces category theory as a solution to faceted filtering. I understood it in theory and it was still does not seem straight forward to implement but I haven't attempted yet. 0. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5145200/#!po=28.8732 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5145200/#!po=28...
- hardwaresofton 4y ago
  So for client-side search, I generally know of Lunr.js: https://lunrjs.com/docs/index.html https://lunrjs.com/docs/index.html There are some others but I can't find them at this moment -- a bunch of the other projects I find are somewhat abandoned, lunr is actually on my list of things to use (because it makes the most sense to just ship a pre-built index with the first like... 5 letters maybe of typeahead, no matter how fast the backend is)
  - Bilal_io 4y ago
    Thanks for the link. This unfortunately is not what I am looking for. Faceted filters are a different beast.
francoismassot 4y ago
You can consider also lnx that is based on tantivy and is performing quite well (https://lnx.rs/ https://lnx.rs/).
nightpool 4y ago
> and if someone is searching on your site excessively maybe there's a problem with your UX (unless you're a search engine or repository of information). I don't understand this comment. Why would you search something that *isn't*, in some senses, a repository of information? I would say almost every website needs to have search in some sense, and it's *because* sites function as a repository of information that they need this search. Think about e.g. Stripe's documentation, or Github's repository / code search. HN is also another great example—I search for stories or comments all the time to try and remember something I read about recently or heard about last week, but couldn't quite remember. I'm hard-pressed to think of a web site I use regularly that *shouldn't* have full-text search, if I'm being honest.
- hardwaresofton 4y ago
  I don't consider use cases like documentation a "repository" of information, but maybe this is just me not phrasing it badly. In the literal sense sure it is, but when I think of a "repository of information" I think of wikipedia, amazon search items, etc. The scale of a documentation site is a very different problem -- you can brute force it in ways that you can't at larger scales. I agree that HN would be a case of the large repository, but even then what most people want out of HN search is pretty simple/basic keyword search. I think a decent non-frustrating HN search feature could be very basic and get by without most of the advanced features/rabbit holes available in search. Basically I think most apps fall into the lighter search use case -- command palettes, search inside of apps with a small scale of information, etc. My comment wasn't that apps shouldn't have full text search -- it was that most that have full text search don't need complex full text search with all the bells and whistles that lucene and other serious search engines provide. These up-and-comers might be enough for a bunch of apps for which search is not the main feature.
  - nightpool 4y ago
    I think the sibling comment on this thread kind of disproves this to some extent. Sure, some simple in-memory search library might be good enough for a command palette or for "apps", but the fact that even tiny docs and "marketing" sites that try to implement site search are almost always outdone by google "site:example.com query" really goes to show how much value a full stemming / synonym clustering / syntax normalizing search engine can bring.
MobiusHorizons 4y ago
Would it make sense to include Sqlite FTS5 in that mix?
- hardwaresofton 4y ago
  It would, I did for the supabase post but... This is already way too much! I have no idea when I'll actually be able to get to all this as-is. Waiting for meilisearch to ingest documents right now and the Show HN is going up.
hardwaresofton 4y ago
Meili is still ingesting documents but we're live: https://news.ycombinator.com/item?id=33321268 https://news.ycombinator.com/item?id=33321268 Maybe I should have used their batch thing instead.
jojo_ 4y ago
Have you heard about Xapian?
- hardwaresofton 4y ago
  Nope I hadn't, but it was mentioned here: https://news.ycombinator.com/item?id=33318533 https://news.ycombinator.com/item?id=33318533

hawski 4y ago

Thank you for this comparison. I would also like to know how Bleve Search (https://github.com/blevesearch/bleve https://github.com/blevesearch/bleve) turns out. I have for many years now a small search engine project in my free-time pipeline, but I'm before crawling even and I intend to sit for searching part after some of that.

hardwaresofton 4y ago
You're right I should put bleve on there as well. This isn't even the whole list. Toshi (https://github.com/toshi-search/Toshi https://github.com/toshi-search/Toshi) is also out there...
- snikolaev 4y ago
  If you decide to add Manticore Search to the list feel free to ping me at sergey@manticoresearch.com if you need help with preparing the ingestion scripts etc.
  - hardwaresofton 4y ago
    Oh! Damn it I forgot about manticore -- I had seen it before but forgot to include it. Eventually all of these projects will be highlighted on Awesome F/OSS (https://awsmfoss.com https://awsmfoss.com), but for now I'm just going to dump my bookmarks here for other people, since I'm leaving awesome projects out: Search Engines AWS OpenSearch https://github.com/opensearch-project/OpenSearch https://github.com/opensearch-project/OpenSearch https://github.com/opensearch-project/OpenSearch-Dashboards https://github.com/opensearch-project/OpenSearch-Dashboards https://github.com/opensearch-project/perftop https://github.com/opensearch-project/perftop https://github.com/go-ego/riot https://github.com/go-ego/riot https://groonga.org/ https://groonga.org/ https://github.com/groonga/groonga https://github.com/groonga/groonga https://github.com/meilisearch/MeiliSearch https://github.com/meilisearch/MeiliSearch https://github.com/mosuka/bayard https://github.com/mosuka/bayard https://github.com/nezaboodka/nevod https://github.com/nezaboodka/nevod https://github.com/searx/searx https://github.com/searx/searx https://github.com/stryku/okon https://github.com/stryku/okon https://github.com/toshi-search/Toshi https://github.com/toshi-search/Toshi https://github.com/typesense/typesense https://github.com/typesense/typesense https://github.com/valeriansaliou/sonic https://github.com/valeriansaliou/sonic Algolia https://github.com/marconi1992/algolite https://github.com/marconi1992/algolite https://quickwit.io/ https://quickwit.io/ https://github.com/quickwit-inc/quickwit https://github.com/quickwit-inc/quickwit https://docs.meilisearch.com/ https://docs.meilisearch.com/ https://github.com/prabhatsharma/zinc https://github.com/prabhatsharma/zinc phalanx https://github.com/blugelabs/bluge https://github.com/blugelabs/bluge https://github.com/mosuka/phalanx https://github.com/mosuka/phalanx https://github.com/mosuka/blast https://github.com/mosuka/blast ManticoreSearch https://github.com/manticoresoftware/manticoresearch https://github.com/manticoresoftware/manticoresearch https://github.com/manticoresoftware/docker https://github.com/manticoresoftware/docker https://manticoresearch.com/blog/manticore-alternative-to-elasticsearch/ https://manticoresearch.com/blog/manticore-alternative-to-el... https://manticoresearch.com/ https://manticoresearch.com/ https://manual.manticoresearch.com/Introduction https://manual.manticoresearch.com/Introduction https://forum.manticoresearch.com/t/manticore-search-cheatsheets/1049 https://forum.manticoresearch.com/t/manticore-search-cheatsh... https://forum.manticoresearch.com/ https://forum.manticoresearch.com/ Whoosh https://whoosh.readthedocs.io/en/latest/ https://whoosh.readthedocs.io/en/latest/ https://pypi.org/project/Whoosh/ https://pypi.org/project/Whoosh/ lyra https://github.com/nearform/lyra https://github.com/nearform/lyra https://nearform.github.io/lyra/ https://nearform.github.io/lyra/ https://github.com/LyraSearch/lyra https://github.com/LyraSearch/lyra https://lyrasearch.io/ https://lyrasearch.io/ flexsearch https://github.com/nextapps-de/flexsearch#performance-benchmark-ranking https://github.com/nextapps-de/flexsearch#performance-benchm... https://pagefind.app/docs/ https://pagefind.app/docs/ Lucene https://github.com/apache/lucene https://github.com/apache/lucene https://lucene.apache.org/ https://lucene.apache.org/ ZincSearch https://zincsearch.com/ https://zincsearch.com/ Solr https://solr.apache.org/ https://solr.apache.org/ https://solr.apache.org/operator/ https://solr.apache.org/operator/ https://solr.apache.org/guide/solr/latest/getting-started/solr-tutorial.html https://solr.apache.org/guide/solr/latest/getting-started/so... https://github.com/apache/solr https://github.com/apache/solr https://solr.apache.org/guide/solr/latest/deployment-guide/solr-in-docker.html https://solr.apache.org/guide/solr/latest/deployment-guide/s... Konnu https://gitlab.com/shadowislord/konnu https://gitlab.com/shadowislord/konnu Quickwit QuickWit + Clickhouse https://clickhouse.com/docs/en/guides/developer/full-text-search/ https://clickhouse.com/docs/en/guides/developer/full-text-se... https://clickhouse.com/docs/en/sql-reference/functions/string-search-functions/ https://clickhouse.com/docs/en/sql-reference/functions/strin... There is no way I can get to running all of these (this project was supposed to be quick!!), but I will run the ones I noted earlier, and probably manticore too since it was high on my list since it's quite polished looking.

Bilal_io 4y ago

Hey that's a great list of tools. Are you aware of any that can be used client side like Lyra and supports faceted search? I've been looking for a solution and cannot find it, even an algorithm and/or a data structure can be helpful. I attempted coming up with a solution myself but ended up with frustration when it came to making the facets dynamic and update as other filters are applied. I read a couple of papers and one stood out [0], which introduces category theory as a solution to faceted filtering. I understood it in theory and it was still does not seem straight forward to implement but I haven't attempted yet. 0. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5145200/#!po=28.8732 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5145200/#!po=28...

hardwaresofton 4y ago
So for client-side search, I generally know of Lunr.js: https://lunrjs.com/docs/index.html https://lunrjs.com/docs/index.html There are some others but I can't find them at this moment -- a bunch of the other projects I find are somewhat abandoned, lunr is actually on my list of things to use (because it makes the most sense to just ship a pre-built index with the first like... 5 letters maybe of typeahead, no matter how fast the backend is)
- Bilal_io 4y ago
  Thanks for the link. This unfortunately is not what I am looking for. Faceted filters are a different beast.

francoismassot 4y ago

You can consider also lnx that is based on tantivy and is performing quite well (https://lnx.rs/ https://lnx.rs/).

nightpool 4y ago

> and if someone is searching on your site excessively maybe there's a problem with your UX (unless you're a search engine or repository of information). I don't understand this comment. Why would you search something that *isn't*, in some senses, a repository of information? I would say almost every website needs to have search in some sense, and it's *because* sites function as a repository of information that they need this search. Think about e.g. Stripe's documentation, or Github's repository / code search. HN is also another great example—I search for stories or comments all the time to try and remember something I read about recently or heard about last week, but couldn't quite remember. I'm hard-pressed to think of a web site I use regularly that *shouldn't* have full-text search, if I'm being honest.

hardwaresofton 4y ago
I don't consider use cases like documentation a "repository" of information, but maybe this is just me not phrasing it badly. In the literal sense sure it is, but when I think of a "repository of information" I think of wikipedia, amazon search items, etc. The scale of a documentation site is a very different problem -- you can brute force it in ways that you can't at larger scales. I agree that HN would be a case of the large repository, but even then what most people want out of HN search is pretty simple/basic keyword search. I think a decent non-frustrating HN search feature could be very basic and get by without most of the advanced features/rabbit holes available in search. Basically I think most apps fall into the lighter search use case -- command palettes, search inside of apps with a small scale of information, etc. My comment wasn't that apps shouldn't have full text search -- it was that most that have full text search don't need complex full text search with all the bells and whistles that lucene and other serious search engines provide. These up-and-comers might be enough for a bunch of apps for which search is not the main feature.
- nightpool 4y ago
  I think the sibling comment on this thread kind of disproves this to some extent. Sure, some simple in-memory search library might be good enough for a command palette or for "apps", but the fact that even tiny docs and "marketing" sites that try to implement site search are almost always outdone by google "site:example.com query" really goes to show how much value a full stemming / synonym clustering / syntax normalizing search engine can bring.

MobiusHorizons 4y ago

Would it make sense to include Sqlite FTS5 in that mix?

hardwaresofton 4y ago
It would, I did for the supabase post but... This is already way too much! I have no idea when I'll actually be able to get to all this as-is. Waiting for meilisearch to ingest documents right now and the Show HN is going up.

hardwaresofton 4y ago

Meili is still ingesting documents but we're live: https://news.ycombinator.com/item?id=33321268 https://news.ycombinator.com/item?id=33321268 Maybe I should have used their batch thing instead.

jojo_ 4y ago

Have you heard about Xapian?

hardwaresofton 4y ago
Nope I hadn't, but it was mentioned here: https://news.ycombinator.com/item?id=33318533 https://news.ycombinator.com/item?id=33318533