4 ms·
This is really needed in the NLP world. Most things are english only.
by EmilStenstrom 8y ago
This is really needed in the NLP world. Most things are english only.
- afsina 8y agoGood luck with Turkish. A very large lookup table may work for %90 cases but it will still fail. A basic `true` lemmatizer requires either a complex graph with rules, or an FST generated from it. input is searched through the graph and morphological disambiguator must be applied to result to pick the correct lemma.
- DFHippie 8y agoOr Welsh. You want a canonical form for "wnaethpwyd"? Try "gwneud"! Some regexes and an exceptions list isn't going to cut it. The more a language needs a lemmatizer for NLP the harder it is to write it.
- tomjakubowski 8y agoCould someone explain what's hard in particular about canonical forms in Welsh?
- wyldfire 8y agospaCy excels here. English, German, Portuguese, and more.
- slx26 8y agoI can personally suggest FreeLing [nlp.lsi.upc.edu/freeling]. It supports most languages in the iberian peninsula (spanish, portuguese, catalan (including multiple dialects), galician), and then it also includes english, italian, french, german, russian, croatian and slovene. It's also more flexible than other tools if you need to go outside those languages and customize something.