5 ms·

Last time I looked, Postgres full text search doesn't really do CJK without third party parsers, which can be a bit of a deal breaker.

zkirill 4y ago
I love Postgres but lack of CJK full-text search is the main reason why I am going with Lucene/Solr for in-app search. It's possible to make Postgres do amazing things (e.g. job queue with SKIP LOCKED!) but at what point do you draw the line and use the right tool for the job?
- blowski 4y ago
  Any idea how well CJK works in general with Postgres? Or is it just on full text where it's lacking?
  - mistrial9 4y ago
    the Japanese national phone company NTT is a major postgresql contributor
  - zkirill 4y ago
    In general, we have absolutely no problem with CJK in Postgres. Search is very application specific and some of our users want to do things like kana + kanji search. [1] I don't even know where you would start with that in Postgres. We also need stuff like language detection and analysis of mixed language data. A big advantage of going with a dedicated search tool is that it teaches you what you don't know about search, and it turned out that we knew pretty much nothing. [1] https://www.elastic.co/blog/implementing-japanese-autocomplete-suggestions-in-elasticsearch https://www.elastic.co/blog/implementing-japanese-autocomple...
  - deleted 4y ago
    [deleted]
- chucky_z 4y ago
  When the feature doesn't exist! For existing workflows/functionality; I think using a new tool for a feature that doesn't (and won't for some time) exist is a _perfectly_ good reason to introduce the tool. On top of that you're looking at using something that is mature and well-used, Lucene/Solr. If you wanted to be cheeky you could go ask your CTO if they would foot the bill for someone like 2nd Quadrant to write CJK full-text search into pgsql. ;) I've done this for some features/issues in random OSS projects and if you feel someone else has the same problem I always feel paying money to have it implemented is well worth it.
- deleted 4y ago
  [deleted]
canadiantim 4y ago
I believe you can do pretty comprehensive CJK search with PGroonga extension for Postgres.
robertsdionne 4y ago
You can use the pg_trgm extension.

zkirill 4y ago

I love Postgres but lack of CJK full-text search is the main reason why I am going with Lucene/Solr for in-app search. It's possible to make Postgres do amazing things (e.g. job queue with SKIP LOCKED!) but at what point do you draw the line and use the right tool for the job?

blowski 4y ago
Any idea how well CJK works in general with Postgres? Or is it just on full text where it's lacking?
- mistrial9 4y ago
  the Japanese national phone company NTT is a major postgresql contributor
- zkirill 4y ago
  In general, we have absolutely no problem with CJK in Postgres. Search is very application specific and some of our users want to do things like kana + kanji search. [1] I don't even know where you would start with that in Postgres. We also need stuff like language detection and analysis of mixed language data. A big advantage of going with a dedicated search tool is that it teaches you what you don't know about search, and it turned out that we knew pretty much nothing. [1] https://www.elastic.co/blog/implementing-japanese-autocomplete-suggestions-in-elasticsearch https://www.elastic.co/blog/implementing-japanese-autocomple...
- deleted 4y ago
  [deleted]
chucky_z 4y ago
When the feature doesn't exist! For existing workflows/functionality; I think using a new tool for a feature that doesn't (and won't for some time) exist is a _perfectly_ good reason to introduce the tool. On top of that you're looking at using something that is mature and well-used, Lucene/Solr. If you wanted to be cheeky you could go ask your CTO if they would foot the bill for someone like 2nd Quadrant to write CJK full-text search into pgsql. ;) I've done this for some features/issues in random OSS projects and if you feel someone else has the same problem I always feel paying money to have it implemented is well worth it.
deleted 4y ago
[deleted]

canadiantim 4y ago

I believe you can do pretty comprehensive CJK search with PGroonga extension for Postgres.

robertsdionne 4y ago

You can use the pg_trgm extension.