5 ms·
While it's certainly good enough for english bodies, its half-assed support for compounds is a problem for bodies in languages that use compounds (German for ex
by pilif 12y ago
While it's certainly good enough for english bodies, its half-assed support for compounds is a problem for bodies in languages that use compounds (German for example).
There is sorta-support for compounds, but only really for ispell dictionaries and the ispell format isn't very good at dealing with compounds (you have to declare all permutations and manually flag them as compoundable) plus the world overall has moved over to hunspell, so even just getting an ispell dictionary is tricky.
As a reminder: This is about finding the "wurst" in "weisswürste" for example.
Furthermore, another problem is that the decision whether the output of a dictionary module should be chained into the next module specified or not is up to the C code of the module itself, not part of the FTS config.
This bit me for example when I wanted to have a thesaurus match for common misspellings and colloquial terms which I then further wanted to feed into the dictionary, again for compound matching.
Unfortunately, the thesaurus module is configured as a non-chainable one, so once there's a thesaurus match, that's what's ending up in the index. No chance to ever also looking it up in the dictionary.
Changing this requires changing the C code and subsequently deploying your custom module, all of which is certainly doable but also additional work you might not want to have to do.
And finally: If you use a dictionary, keep in mind that it's by default not shared between connections. That means that you have to pay the price of loading that dictionary whenever you use the text search engine the first time over a connection.
For smaller dictionaries, this isn't significant, but due to the compound handling in ispell, you'll have huge-ass(tm) dictionaries. Case in point is ours which is about 25 Megs in size and costs 0.5 seconds of load time on an 8 drive 15K RAID10 array.
In practice (i.e. whenever you want to respond to a search in less than 0.5 secs, which is, I would argue, always), this forces you to either use some kind of connection pooling (which can have other side-effects for your application), or you use the shared_ispell extension (http://pgxn.org/dist/shared_ispell http://pgxn.org/dist/shared_ispell), though I've seen crashes in production when using that which led me to go back to pgbouncer.
Aside of these limitations (neither of which will apply to you if you have an english body, because searching these works without a dictionary to begin with), yes, it works great.
(edited: Added a note about chaining dictionary modules)
- donw 12y agoI've got a project that needed multilingual support over very, very small documents, and ended up just rolling that myself -- neither ElasticSearch nor Postgres cope very well when you start mixing multiple languages in the same document.
- pilif 12y agoIf it's indeed the same document, you're screwed. Yes. But in Postgres, I could at least have different column (or join tables with different language codes) for different languages and then query them with either dictionary. ElasticSearch doesn't support that: All fields in a document are assumed to be of the same language which leads to a huge duplication of documents in cases where most fields of a document are language-independent.
- donw 12y agoThat's basically what I ended up doing -- writing an analyzer that made a reasonable guess as to what language each word was, and filing it into the appropriate bucket, with a generic bucket for "everything else". Ranking is another story, though.
- izacus 12y agoI don't know about ElasticSearch, but the more popular alternative Solr, can do that easily by using multiple cores (each configured and tweaked for each language) and doing sharding queries, which merge results for you.
- jrochkind1 12y agoyou don't need to use multiple cores for that, although you could -- but in general, anything you can do in multiple cores you can do identically in one core, the only reason to use multiple cores is for performance when you have so much corpus that one machine can't handle it. But yeah, Solr gives you a lot of flexibility, perhaps more than ElasticSearch (which is both it's pro and it's con, for 'on the path' use cases I get the impression ElasticSearch requires less thinking in configuration than solr. although i have not any direct experience with ElasticSearch).