3 ms·
For morphology (esp. languages with complicated morphologies, like Berber), the tool that most computational linguists reach for is finite state transducers, pa
by mcswell 7y ago
For morphology (esp. languages with complicated morphologies, like Berber), the tool that most computational linguists reach for is finite state transducers, particularly those that are built for use in morphology and phonology. An early one of these was the Xerox xfst/ lexc program, which has since been re-implemented in open source form as Foma (https://fomafst.github.io/ https://fomafst.github.io/). The book on xfst/lexc, https://www.press.uchicago.edu/ucp/books/book/distributed/F/bo3613750.html https://www.press.uchicago.edu/ucp/books/book/distributed/F/..., is probably still the best place to go for a tutorial. Other FST programs that have been used for morphology and phonology include the Stuttgart FST (sfst, https://www.ims.uni-stuttgart.de/forschung/ressourcen/werkzeuge/sfst/ https://www.ims.uni-stuttgart.de/forschung/ressourcen/werkze...) and the Helsinki HFST (http://hfst.github.io/ http://hfst.github.io/). HFST allows the use of weights, which can be useful for spell correction.
I've built built morph parsers with all of these except HFST, although that's next on my list.
I'm not familiar with Delphin, but a quick glance at their website implies that it's for syntax, not so much for morphology. They mention a Japanese grammar implemented in Delphin, but it uses a separate tool for morphology.
In answer to your other question, the last time I looked, machine learning of morph parsers (or stemmers, which are like morph parsers that throw away the affixal information) is reasonably good for "fusional" morphologies, which most modern IndoEuropean languages have. I don't think the state-of-the-art ML would work well for Berber, because of its much more complex morphology.
- Rotten194 7y agoThe Delph-in system has a morphological model based on position classes. Morphology items are treated similarly to syntax items, in that they can hold constraints on & apply features to what they attach.
- mcswell 7y agoInteresting, I hadn't seen that. Does it handle phonologically-conditioned allomorphy or inflection classes? Stem allomorphy conditioned by phonology or position in the paradigm (like the stem allomorphs of the Spanish verbs 'tener', 'crecer' etc.)?