3 ms·
FTS in all databases suck for non-English languages. Even the larger dedicated search systems often fail at even the simplest searches or need arcane configurat
by troupo 3y ago
FTS in all databases suck for non-English languages. Even the larger dedicated search systems often fail at even the simplest searches or need arcane configurations to work properly.
- withinboredom 3y agoThey mostly suck because non-english speakers don't contribute to the "magic" like stems and stop-words. It isn't the databases fault it doesn't have good default data.
- giovannibonetti 3y agoTo overcome the need for those arcane configurations, semantic search with vectors from text embeddings are slowly getting traction. Here [1] is an article that shows how to do that with the modern Postgres extension pgvector [2]. The downside is the reliance on a ML model to generate the embeddings, either from OpenAI as mentioned in the article from Supabase, or from an open-source library. [1] https://supabase.com/blog/openai-embeddings-postgres-vector https://supabase.com/blog/openai-embeddings-postgres-vector [2] https://github.com/pgvector/pgvector https://github.com/pgvector/pgvector