4 ms·
the biggest issue is postgres not having tf-idf or BM25 style relevance algorithms. that would have been the right fit for fuzzy name matching.
by sandGorgon 6y ago
the biggest issue is postgres not having tf-idf or BM25 style relevance algorithms.
that would have been the right fit for fuzzy name matching.
- jimktrains2 6y agoHow would tf-idf help fuzzy match names? You would first need to decide the term youre looking for is the term you've indexed, which is what soundex &c would do.
- fumblebee 6y agoSoundex is a useful option for reducing the search space (the blocking stage, with aims to have a high recall). TF-IDF is a useful option as part of a high precision pipeline _post_ blocking. A two step pipeline is typical and necessary because the high precision pipeline, while more accurate than the blocking algorithm, suffers from being slow.
- sandGorgon 6y agoI work in non-English markets. Soundex is not a general replacement. Similaritiy search (https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html https://www.elastic.co/guide/en/elasticsearch/reference/curr...) is far better assuming you have some corpus already.
- michelpp 6y agoTF-IDF algorithms like BM25 do not come built in, but are straightforward to implement on top of postgres' text search: https://www.alibabacloud.com/blog/keyword-analysis-with-postgresql-cosine-and-linear-correlation-algorithms-for-text-analysis_595793 https://www.alibabacloud.com/blog/keyword-analysis-with-post... The key is you have to compute the IDF into a side table using the ts_stat function, since this is a global property (inter-document frequency) of your corpus. Once you have your IDF table you can write a ranking function based on that table.
- sandGorgon 6y agoperformance. that's the issue. the Alibaba Cloud article is nice..but we are really comparing this with Elastic