6 ms·
I would add: you should look for alternative solutions when you need to search anything other than English.
by troupo 2y ago
I would add: you should look for alternative solutions when you need to search anything other than English.
- indulona 2y agoEh, why? Have you ever heard of transliteration?
- troupo 2y agoBecause search in different languages needs to be aware of that language's peculiarities. Including but not limited to things that English doesn't have or only has in vestigial forms like grammatical cases, complex word morphology, declensions etc.
- deleted 2y ago[deleted]
- fpoling 2y agoFor example, it took Google years before they properly implemented search in Russian. That allowed Russian Yandex to win Russian market.
- Terr_ 2y agoBack atcha: Have you ever heard of stemming? That'll be different depending on the language, you can't simply transliterate words to another language, the mapping may change things and the rules might be different. For example, in English "manage / managers", are practically the same, but transliterating to Spanish can give you "gestionan / gerentes", which diverge very early on.
- pezezin 2y agoAlternatively you could use "gestor", which has the same stem. "Gestionar" and "gerente" have different origins, so no wonder they look very different. But yes, most (all?) European languages have vastly more complex morphology than English.
- kiwicopple 2y agopgroonga is a good Postgres option in these cases https://supabase.com/docs/guides/database/extensions/pgroonga https://supabase.com/docs/guides/database/extensions/pgroong...
- troupo 2y agoIt's a bit weird that the pages claim to support all languages, and all examples are in English. Even the documentation for Groonga, on which it's based doesn't talk about multiple languages. It looks like at best it makes Postgres aware of more characters
- jillesvangurp 2y agoOr when you care at all about optimizing your search ranking. Postgresql has a few alright tools to do vector search and to do some simple things with trigrams. And that can work well for some narrow use cases. If that's all you need, great. But this doesn't give you a whole lot of control over search ranking. And if search is in anyway on the critical path to revenue for whatever it is your product does, you probably need to invest in making sure your users find what they need. And that means getting a bit systematic about things, using specialized tools, and making sure you have some skills on your team that know how to do this.
- 0x008 2y agoYou can use custom ranking functions (shown in the article) add weights to columns, etc. What techniques are you talking about that cannot be implemented in Postgres?
- jillesvangurp 2y agoThe hundreds of features things like Solr and Elasticsearch/Opensearch have. These are complex products but mostly this is a form of necessary complexity. To be blunt, the intersection of people that know what they are doing on this front and people that choose to use Postgres for this is pretty narrow. It does happen and I've seen a few nice things built with Postgres. But mostly it's just people using the wrong tool for the job.
- danielheath 2y agoI agree with you, but to be fair, that's true of virtually every technology I've encountered in my career.
- philippemnoel 2y agoThis is partially true. You're right that the feature set is necessary, but it's now possible to do this in Postgres. ParadeDB pg_search bundles Tantivy, a Lucene-inspired library inside Postgres, to give users the feature set (BM25 ranking, tokenizers, faceted search, etc.) while keeping the benefits of Postgres (keep your data normalized, avoid ETL, etc.) Disclaimer: I work on ParadeDB
- 0x008 2y agoBut tsvector supports all sorts of different languages, at least western.
- philippemnoel 2y agoParadeDB pg_search supports tokenizers in all major languages
- troupo 2y agoGood find/tip! It took me a while to find it, it looks like they support it via Snowball: https://github.com/snowballstem/snowball https://github.com/snowballstem/snowball I wish ParadeDB exposed multi-language search capability more prominently in the docs.